Nov 02 17:22:06.688608 ip-10-0-129-72 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:22:07.074467 ip-10-0-129-72 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:07.074467 ip-10-0-129-72 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:22:07.074467 ip-10-0-129-72 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:07.074467 ip-10-0-129-72 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:07.077592 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077512    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.077592 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077593    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077598    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077601    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077604    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077607    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077609    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077612    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077615    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077617    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077620    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077622    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077625    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077627    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077634    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077637    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077639    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077642    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077644    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077647    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077650    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.077658 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077652    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077654    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077657    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077659    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077662    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077665    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077668    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077670    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077673    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077675    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077678    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077680    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077682    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077685    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077687    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077689    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077692    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077694    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077697    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.078214 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077700    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077702    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077705    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077707    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077709    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077712    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077714    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077716    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077719    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077721    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077724    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077726    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077729    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077731    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077734    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077736    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077739    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077741    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077744    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077747    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.078669 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077749    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077752    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077754    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077757    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077760    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077762    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077765    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077767    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077770    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077772    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077775    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077777    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077780    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077783    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077785    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077789    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077793    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077796    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077798    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.079124 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077801    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077804    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077807    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077809    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077812    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077814    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077817    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077822    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077826    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077829    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077832    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077834    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077837    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077839    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077842    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077844    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077846    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077849    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077852    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.079602 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077854    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077856    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077859    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077861    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077864    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077866    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077869    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.077874    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079326    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079335    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079338    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079341    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079343    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079346    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079348    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079352    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079356    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079358    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079361    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079364    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.080038 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079367    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079369    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079372    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079375    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079377    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079380    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079383    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079386    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079388    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079391    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079395    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079398    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079401    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079404    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079406    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079409    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079411    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079414    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079417    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.080536 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079422    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079425    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079428    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079431    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079434    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079436    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079439    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079441    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079445    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079448    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079451    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079453    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079456    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079458    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079461    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079463    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079466    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079469    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079471    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079482    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.080999 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079485    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079488    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079491    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079493    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079496    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079498    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079501    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079503    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079506    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079508    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079529    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079532    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079535    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079538    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079541    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079543    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079546    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079548    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079551    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.081470 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079554    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079557    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079560    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079562    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079565    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079567    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079570    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079573    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079575    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079578    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079580    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079583    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079585    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079588    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079590    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079593    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079595    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079597    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079600    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079602    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.081904 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079605    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079608    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079611    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079614    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079616    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079620    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079623    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079625    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079628    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079631    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079633    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079636    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079638    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079641    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079643    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.079646    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080143    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080148    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080151    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.082389 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080153    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080156    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080158    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080161    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080163    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080169    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080172    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080174    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080177    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080179    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080182    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080184    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080187    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080190    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080192    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080194    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080197    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080200    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.082840 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080202    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080205    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080208    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080211    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080214    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080216    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080219    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080222    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080224    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080227    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080230    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080232    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080235    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080237    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080240    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080242    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080245    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080247    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080249    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.083283 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080252    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080254    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080257    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080273    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080275    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080278    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080281    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080283    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080286    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080289    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080291    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080294    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080297    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080300    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080302    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080305    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080309    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080313    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080317    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080321    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.083759 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080324    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080326    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080329    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080332    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080334    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080337    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080339    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080342    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080344    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080347    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080349    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080352    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080354    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080357    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080359    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080362    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080365    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080367    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080370    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080373    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.084442 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080375    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080378    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080380    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080383    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080385    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080388    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080390    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080393    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080395    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080399    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080401    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080403    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080406    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080408    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080411    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080413    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080416    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080418    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080421    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080423    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.084931 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080427    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080430    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080435    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080438    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.080441    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080550    2578 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080561    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080570    2578 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080576    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080580    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080583    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080588    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080605    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080611    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080614    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080618    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080621    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080624    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080627    2578 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080630    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080633    2578 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080635    2578 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080639    2578 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:22:07.085434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080644    2578 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080647    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080650    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080653    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080656    2578 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080660    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080663    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080666    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080669    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080671    2578 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080674    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080677    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080680    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080683    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080687    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080690    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080693    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080697    2578 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080700    2578 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080703    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080707    2578 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080710    2578 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080714    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080718    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:22:07.085953 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080721    2578 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080725    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080728    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080731    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080734    2578 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080737    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080740    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080743    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080747    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080750    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080752    2578 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080755    2578 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080759    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080762    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080765    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080768    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080771    2578 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080774    2578 flags.go:64] FLAG: --help="false"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080777    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080780    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080783    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080785    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080789    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:22:07.086531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080792    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080795    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080799    2578 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080802    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080805    2578 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080808    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080811    2578 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080815    2578 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080818    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080821    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080824    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080827    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080830    2578 flags.go:64] FLAG: --lock-file=""
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080833    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080836    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080839    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080844    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080847    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080851    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080853    2578 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080856    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080860    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080862    2578 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080865    2578 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080869    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:22:07.087130 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080872    2578 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080876    2578 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080879    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080882    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080884    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080887    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080890    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080893    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080896    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080907    2578 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080910    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080913    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080916    2578 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080919    2578 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080923    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080926    2578 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080929    2578 flags.go:64] FLAG: --port="10250"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080932    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080935    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0eb094e71a9f34389"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080938    2578 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080940    2578 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080943    2578 flags.go:64] FLAG: --register-node="true"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080946    2578 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080950    2578 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080953    2578 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080956    2578 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:22:07.088231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080960    2578 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080964    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080967    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080970    2578 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080972    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080975    2578 flags.go:64] FLAG: --runonce="false"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080978    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080980    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080984    2578 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080986    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080989    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080992    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080995    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.080998    2578 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081001    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081005    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081008    2578 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081011    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081014    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081017    2578 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081021    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081026    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081029    2578 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081032    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081037    2578 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:22:07.089346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081040    2578 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081043    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081046    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081049    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081052    2578 flags.go:64] FLAG: --v="2"
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081056    2578 flags.go:64] FLAG: --version="false"
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081060    2578 flags.go:64] FLAG: --vmodule=""
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081065    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081069    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081202    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081206    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081209    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081212    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081214    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081217    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081219    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081222    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081224    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081226    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081229    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081231    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081234    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.090081 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081237    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081240    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081243    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081245    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081248    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081251    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081254    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081256    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081259    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081275    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081277    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081279    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081282    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081285    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081287    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081290    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081293    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081295    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081297    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081301    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.090831 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081303    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081306    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081309    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081311    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081314    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081317    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081319    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081322    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081324    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081326    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081329    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081331    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081335    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081338    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081340    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081342    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081345    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081349    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081351    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081354    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.091822 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081356    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081359    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081361    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081364    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081366    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081369    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081371    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081375    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081378    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081380    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081383    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081385    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081388    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081390    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081393    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081396    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081398    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081401    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081403    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.092662 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081406    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081408    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081411    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081413    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081416    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081419    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081422    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081424    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081427    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081429    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081433    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081435    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081438    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081440    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081442    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081445    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081447    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081449    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081452    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.093236 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081454    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081458    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081462    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081465    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081468    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081470    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081473    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081476    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081479    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081482    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081484    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081486    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081489    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081491    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081494    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.093995 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.081519    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081555    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081565    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081570    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081573    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081576    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081578    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081582    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081586    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081590    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081593    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081595    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081598    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081601    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081603    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081605    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081608    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081610    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081612    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.094546 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081615    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081617    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081620    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081622    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081625    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081627    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081631    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081634    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081636    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081638    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081641    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081643    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081645    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081648    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081650    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081653    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081657    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081659    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081662    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081664    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.095109 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081667    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081669    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081673    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081675    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081678    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081680    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081683    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081685    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081687    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081690    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081692    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081695    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081697    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081700    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081702    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081705    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081707    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081709    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081712    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081715    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.095746 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081718    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081721    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081724    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081726    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081728    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081731    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081733    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081735    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081739    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081741    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081744    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081746    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081748    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081751    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081754    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081757    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081759    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081762    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081764    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.096472 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081767    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081769    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081771    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081775    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081778    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081781    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081783    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081786    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081788    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081791    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081793    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081796    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081798    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081808    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081810    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081813    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081816    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081818    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081821    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.096957 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081823    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081826    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081830    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081832    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081835    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081838    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081840    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081843    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081847    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.081850    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.089567    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.089589    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.090384    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.090396    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090486    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090494    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090499    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090505    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090509    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090514    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.097518 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090519    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090523    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090528    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090532    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090536    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090541    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090545    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090549    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090553    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090557    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090562    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090566    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090570    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090574    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090578    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090582    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090587    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090590    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090595    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090599    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.098093 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090603    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090608    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090612    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090618    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090623    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090627    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090631    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090635    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090639    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090644    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090648    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090652    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090656    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090662    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090668    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090674    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090678    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090683    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090688    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.098745 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090692    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090696    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090700    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090704    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090709    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090713    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090717    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090722    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090726    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090730    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090734    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090738    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090742    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090746    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090750    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090755    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090760    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090765    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090770    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090774    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.099373 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090778    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090782    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090786    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090790    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090794    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090798    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090802    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090806    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090811    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090815    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090819    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090822    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090826    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090830    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090834    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090838    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090842    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090846    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090851    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.099968 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090855    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090859    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090865    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090871    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090876    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090881    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090885    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090890    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090894    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090899    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090903    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090908    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090913    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090917    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090921    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090925    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090929    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090933    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090937    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.100532 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090941    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090947    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.090951    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.090993    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091294    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091303    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091307    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091311    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091315    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091319    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091323    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091327    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091331    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091335    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091339    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.101052 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091342    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091346    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091350    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091354    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091358    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091362    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091367    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091371    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091376    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091379    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091385    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091389    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091393    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091396    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091400    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091403    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091408    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091412    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091416    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.101426 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091420    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091426    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091432    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091436    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091442    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091446    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091452    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091456    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091461    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091466    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091470    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091485    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091489    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091493    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091497    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091501    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091505    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091529    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091533    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.102056 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091537    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091542    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091546    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091550    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091555    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091561    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091566    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091570    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091574    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091579    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091583    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091587    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091592    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091596    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091600    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091604    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091609    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091613    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091617    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091621    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.102587 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091625    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091629    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091634    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091638    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091642    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091646    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091650    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091654    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091658    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091662    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091667    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091671    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091675    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091680    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091684    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091689    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091693    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091698    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091703    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091708    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.103051 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091712    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091716    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091721    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091725    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091729    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091733    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091740    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091746    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091750    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091755    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091759    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091763    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091767    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091772    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091776    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091780    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.103622 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091784    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.091830    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091883    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091890    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091895    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091900    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091904    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091911    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091917    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091922    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091927    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091931    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091935    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091939    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091943    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:07.104016 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091949    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091953    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091957    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091961    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091965    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091969    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091973    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091977    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091982    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091986    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091990    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091994    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.091998    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092002    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092006    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092010    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092014    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092018    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092022    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092027    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:07.104398 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092031    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092035    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092039    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092043    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092048    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092052    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092056    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092060    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092065    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092069    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092073    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092077    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092081    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092085    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092089    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092093    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092097    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092101    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092105    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092109    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:07.104885 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092113    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092117    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092125    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092129    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092133    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092138    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092142    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092146    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092152    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092156    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092161    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092165    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092169    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092173    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092177    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092181    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092188    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092195    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092200    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:07.105463 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092204    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092208    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092212    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092216    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092220    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092225    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092229    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092234    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092238    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092243    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092247    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092251    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092255    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092259    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092281    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092287    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092291    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092295    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092299    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092304    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:07.105926 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092308    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092312    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092317    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092321    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092325    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092331    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092336    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092340    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092345    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092349    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092353    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092357    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092361    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:07.092365    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.092451    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.094902    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.097458    2578 server.go:1014] "Starting client certificate rotation"
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.097572    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:22:07.106647 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.097626    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:22:07.107084 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.100327    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:07.107084 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.101473    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:07.110456 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.110439    2578 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:22:07.113412 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.113397    2578 log.go:25] "Validated CRI v1 image API"
Nov 02 17:22:07.114212 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.114191    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:22:07.119865 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.119844    2578 fs.go:135] Filesystem UUIDs: map[1ffc1acf-636e-48b2-9474-2339c4325f09:/dev/nvme0n1p3 5040071e-edea-48b9-a752-040513b24909:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 17:22:07.119941 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.119864    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 17:22:07.126068 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.125935    2578 manager.go:217] Machine: {Timestamp:2026-11-02 17:22:07.1244996 +0000 UTC m=+0.321766376 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3171537 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25ed9c7d79be194a99120b3e6c72e1 SystemUUID:ec25ed9c-7d79-be19-4a99-120b3e6c72e1 BootID:25b6b15b-6e60-47be-ad3d-de28f10ff2f2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:95:67:ee:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:95:67:ee:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:7c:3f:e4:d5:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:22:07.126834 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.126822    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:22:07.126951 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.126934    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:22:07.130182 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.130150    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:22:07.130339 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.130183    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-72.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:22:07.130383 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.130341    2578 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:22:07.130383 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.130349    2578 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:22:07.130430 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.130389    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:07.130430 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.130408    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:07.131226 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.131216    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:22:07.135048 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.135033    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:22:07.135373 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.135363    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:22:07.136010 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.135999    2578 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:22:07.136049 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.136017    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:22:07.136049 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.136029    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:22:07.136049 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.136038    2578 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:22:07.136049 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.136046    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:22:07.136490 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.136473    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:07.139163 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.139144    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:22:07.145009 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.144993    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:07.146093 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146080    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:22:07.146143 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146111    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:22:07.146303 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146291    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:22:07.146340 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146310    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:22:07.146340 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146317    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:22:07.146340 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146322    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:22:07.146340 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146327    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:22:07.146340 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146333    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:22:07.146775 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146766    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:22:07.146802 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146776    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:22:07.146802 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146784    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:22:07.146802 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146789    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:22:07.146802 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146797    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:22:07.146915 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.146805    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:22:07.147363 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.147353    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:22:07.147394 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.147366    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:22:07.151700 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.151685    2578 server.go:1263] "Started kubelet"
Nov 02 17:22:07.151823 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.151768    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:22:07.151897 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.151849    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:22:07.151944 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.151772    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:22:07.152630 ip-10-0-129-72 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:22:07.153108 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.153050    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:22:07.154720 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.153779    2578 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:22:07.157728 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.157703    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 17:22:07.158752 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.158731    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:22:07.159112 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.159097    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:22:07.159212 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.159194    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-72.us-west-1.compute.internal\" not found"
Nov 02 17:22:07.159609 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.159584    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:22:07.159680 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.159670    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:22:07.159730 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.159686    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:22:07.159890 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.159877    2578 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:22:07.159890 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.159890    2578 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:22:07.160137 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.160120    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:22:07.160137 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.160138    2578 factory.go:55] Registering systemd factory
Nov 02 17:22:07.160247 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.160146    2578 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:22:07.160404 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.160388    2578 factory.go:153] Registering CRI-O factory
Nov 02 17:22:07.160439 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.160409    2578 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:22:07.160439 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.160424    2578 factory.go:103] Registering Raw factory
Nov 02 17:22:07.160439 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.160433    2578 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:22:07.161311 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.161292    2578 manager.go:319] Starting recovery of all containers
Nov 02 17:22:07.172747 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.172619    2578 manager.go:324] Recovery completed
Nov 02 17:22:07.178184 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.178168    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:07.180091 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.180074    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:07.180161 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.180105    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:07.180161 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.180117    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:07.180661 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.180649    2578 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:22:07.180700 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.180660    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:22:07.180700 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.180683    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:22:07.180776 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.180685    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:07.181584 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.181569    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:07.181654 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.181599    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:07.181654 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.181613    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:07.181814 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.181801    2578 policy_none.go:50] "Start"
Nov 02 17:22:07.181878 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.181815    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:22:07.181878 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.181825    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:22:07.182716 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.182707    2578 policy_none.go:44] "Start"
Nov 02 17:22:07.232134 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.232086    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:22:07.236462 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.236445    2578 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:22:07.236522 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.236496    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:22:07.236522 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.236506    2578 server.go:87] "Starting device plugin registration server"
Nov 02 17:22:07.236860 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.236848    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:22:07.236889 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.236864    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:22:07.236975 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.236961    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:22:07.237071 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.237053    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:22:07.237071 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.237066    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:22:07.237674 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.237644    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:22:07.237737 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.237703    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-72.us-west-1.compute.internal\" not found"
Nov 02 17:22:07.295770 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.295735    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-72.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 17:22:07.295770 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.295755    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-72.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 17:22:07.297768 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.295743    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-72.us-west-1.compute.internal.18e44a3149c2cfbc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-72.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-72.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 17:22:07.151656892 +0000 UTC m=+0.348923663,LastTimestamp:2026-11-02 17:22:07.151656892 +0000 UTC m=+0.348923663,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-72.us-west-1.compute.internal,}"
Nov 02 17:22:07.301257 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.301221    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:22:07.302515 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.302498    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:22:07.302559 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.302521    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:22:07.302559 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.302541    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:22:07.302623 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.302571    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 17:22:07.337589 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.337495    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:07.338725 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.338689    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:07.338725 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.338730    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:07.338874 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.338751    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:07.338874 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.338780    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.374276 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.374240    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:22:07.374436 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.374315    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fszt9"
Nov 02 17:22:07.403668 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.403614    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"]
Nov 02 17:22:07.403832 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.403691    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:07.405199 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.405177    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:07.405337 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.405220    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:07.405337 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.405235    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:07.407622 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.407606    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:07.407754 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.407740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.407808 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.407771    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:07.408411 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.408390    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:07.408501 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.408423    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:07.408501 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.408391    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:07.408501 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.408439    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:07.408501 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.408458    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:07.408501 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.408471    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:07.410645 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.410630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.410705 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.410655    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:07.411395 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.411383    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:07.411485 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.411405    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:07.411485 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.411414    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:07.434061 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.434041    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-72.us-west-1.compute.internal\" not found" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.436234 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:07.436215    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-72.us-west-1.compute.internal\" not found" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.454795 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.454774    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:07.456466 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.456448    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:07.462349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.462326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a902dc8a4f9241e56ab9949e623f113-config\") pod \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"7a902dc8a4f9241e56ab9949e623f113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.462481 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.462355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.462481 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.462375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.554420 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.554388    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:07.555073 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.555053    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:07.556345 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.556328    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.559300 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.559284    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.563505 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.563486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a902dc8a4f9241e56ab9949e623f113-config\") pod \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"7a902dc8a4f9241e56ab9949e623f113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.563614 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.563539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.563614 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.563576    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a902dc8a4f9241e56ab9949e623f113-config\") pod \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"7a902dc8a4f9241e56ab9949e623f113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.563614 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.563597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.563614 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.563578    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.563749 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.563618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.673386 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.673308    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:07.674034 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.674021    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.737449 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.737418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.739608 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.739585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:07.775797 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:07.775766    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:08.097319 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.097285    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 17:22:08.098046 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.097462    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:22:08.098046 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.097495    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:22:08.098046 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.097522    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:22:08.098046 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.097511    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://acf688e646e8844ffb176668e16206eb-5f8c15bb6010e1e3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-72.us-west-1.compute.internal.18e44a314b75230d\": read tcp 10.0.129.72:44052->18.220.170.94:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-72.us-west-1.compute.internal.18e44a314b75230d  default   4550 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-72.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-72.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-72.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 17:22:07 +0000 UTC,LastTimestamp:2026-11-02 17:22:07.33875477 +0000 UTC m=+0.536021538,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-72.us-west-1.compute.internal,}"
Nov 02 17:22:08.136222 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.136200    2578 apiserver.go:52] "Watching apiserver"
Nov 02 17:22:08.159559 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.159530    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 17:22:08.378727 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.378635    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:17:08 +0000 UTC" deadline="2028-08-15 21:13:47.29008288 +0000 UTC"
Nov 02 17:22:08.378727 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.378668    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15651h51m38.911419167s"
Nov 02 17:22:08.409934 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.409900    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:22:08.410231 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.410214    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zfl47","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-mj6cq","kube-system/global-pull-secret-syncer-9m4h8","kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal","openshift-multus/multus-586s4","openshift-multus/multus-additional-cni-plugins-g7sgh","kube-system/konnectivity-agent-m7rhm","openshift-network-diagnostics/network-check-target-przls","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff","openshift-image-registry/node-ca-gmfdg","openshift-multus/network-metrics-daemon-h7btw","openshift-ovn-kubernetes/ovnkube-node-455n9","openshift-cluster-node-tuning-operator/tuned-q26xm"]
Nov 02 17:22:08.412784 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.412766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.414774 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.414752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.416706 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.416693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.416771 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.416746    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:08.418638 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.418622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.420678 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.420662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.422664 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.422652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:08.424701 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.424688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:08.424767 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.424753    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:08.426757 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.426735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.429199 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.429176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.431441 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.431425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:08.431565 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.431504    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:08.433690 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.433675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.435824 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.435807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.460441 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.460413    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:22:08.466840 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.466819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-ovn\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.466935 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.466871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-tuned\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.466935 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.466897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cnibin\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.467021 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.466955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.467021 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-kubelet\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.467089 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwczd\" (UniqueName: \"kubernetes.io/projected/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-kube-api-access-bwczd\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.467147 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-log-socket\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467175 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.467215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-registration-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.467249 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvvm\" (UniqueName: \"kubernetes.io/projected/78779265-8cc2-4286-8ad2-3aed34f341dd-kube-api-access-bgvvm\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.467249 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-node-log\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467336 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-bin\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467336 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-netd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467424 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dbg\" (UniqueName: \"kubernetes.io/projected/de90b397-c8f3-4b8e-b89a-822b7bc465bd-kube-api-access-79dbg\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.467424 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52674954-19d1-447d-ade5-d8655b4db351-hosts-file\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.467529 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.467529 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-kubelet\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467529 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-modprobe-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.467687 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-tmp\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.467687 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-agent-certs\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:08.467687 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:08.467687 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-slash\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467806 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-var-lib-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467806 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbrbx\" (UniqueName: \"kubernetes.io/projected/481d442d-4aa0-40bb-b723-ed569f0386ca-kube-api-access-sbrbx\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.467806 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-systemd\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.467939 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.467802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.468070 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-bin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.468123 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-hostroot\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.468184 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-konnectivity-ca\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:08.468233 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.468336 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.468456 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p75t\" (UniqueName: \"kubernetes.io/projected/52674954-19d1-447d-ade5-d8655b4db351-kube-api-access-5p75t\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-multus-certs\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-etc-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/481d442d-4aa0-40bb-b723-ed569f0386ca-ovn-node-metrics-cert\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysconfig\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-dbus\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-socket-dir-parent\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52674954-19d1-447d-ade5-d8655b4db351-tmp-dir\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-sys-fs\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-script-lib\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-sys\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.468985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cni-binary-copy\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-etc-kubernetes\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp25t\" (UniqueName: \"kubernetes.io/projected/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-kube-api-access-dp25t\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.470343 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-system-cni-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8ec8714-849f-4245-b41e-cb84f8a4996f-host\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrvd6\" (UniqueName: \"kubernetes.io/projected/538a6445-5d03-43a2-a41c-0c8e89a53553-kube-api-access-hrvd6\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-systemd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a35c68-067f-4fac-82f0-a7ef888b21fb-iptables-alerter-script\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndvj\" (UniqueName: \"kubernetes.io/projected/34a35c68-067f-4fac-82f0-a7ef888b21fb-kube-api-access-6ndvj\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-k8s-cni-cncf-io\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-kubernetes\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-conf\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-run\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8ec8714-849f-4245-b41e-cb84f8a4996f-serviceca\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-netns\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-lib-modules\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-var-lib-kubelet\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.470888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-kubelet-config\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-system-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-netns\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-multus\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-daemon-config\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a35c68-067f-4fac-82f0-a7ef888b21fb-host-slash\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-conf-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-socket-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-device-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.469977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-systemd-units\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.470004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-config\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.470032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-env-overrides\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.470055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-host\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.471357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.470082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cnibin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.471808 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.470110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2d4\" (UniqueName: \"kubernetes.io/projected/d8ec8714-849f-4245-b41e-cb84f8a4996f-kube-api-access-mz2d4\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.476346 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.476318    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:22:08.476781 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.476766    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkkxw\""
Nov 02 17:22:08.478624 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.478603    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:22:08.478745 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.478734    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:08.479037 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.479025    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:22:08.479380 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.479370    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:22:08.479711 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.479697    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jxqfl\""
Nov 02 17:22:08.481576 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.481555    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:08.481732 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.481714    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:22:08.481823 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.481759    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:22:08.481823 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.481759    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:22:08.481823 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.481765    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l2rxq\""
Nov 02 17:22:08.481823 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.481785    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:22:08.481946 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.481838    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:22:08.490281 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.490252    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wf5x\""
Nov 02 17:22:08.516121 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516049    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:22:08.516280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516246    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:08.516341 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516295    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:22:08.516391 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516348    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:22:08.516391 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516361    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-546w9\""
Nov 02 17:22:08.516391 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516373    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:22:08.516500 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516413    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:22:08.516500 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516427    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:22:08.516500 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516473    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:08.516631 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516561    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4464\""
Nov 02 17:22:08.516684 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516645    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:22:08.516684 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516675    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:22:08.516752 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516682    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:22:08.516752 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.516690    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:22:08.529623 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.529607    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:22:08.530588 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.530566    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:22:08.530588 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.530565    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:22:08.530727 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.530616    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-867zk\""
Nov 02 17:22:08.530727 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.530620    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7kdpj\""
Nov 02 17:22:08.530727 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.530649    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztdmt\""
Nov 02 17:22:08.532826 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.532810    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:22:08.532826 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.532809    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:22:08.532918 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.532857    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkmvf"
Nov 02 17:22:08.570967 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.570941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-dbus\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.570967 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.570972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-socket-dir-parent\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.571160 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.570988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52674954-19d1-447d-ade5-d8655b4db351-tmp-dir\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.571160 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:08.571160 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-sys-fs\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.571160 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571088    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-sys-fs\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.571160 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-socket-dir-parent\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.571357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-script-lib\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.571357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-sys\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.571357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cni-binary-copy\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.571357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-etc-kubernetes\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.571357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp25t\" (UniqueName: \"kubernetes.io/projected/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-kube-api-access-dp25t\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.571357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-system-cni-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.571357 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8ec8714-849f-4245-b41e-cb84f8a4996f-host\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrvd6\" (UniqueName: \"kubernetes.io/projected/538a6445-5d03-43a2-a41c-0c8e89a53553-kube-api-access-hrvd6\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-systemd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52674954-19d1-447d-ade5-d8655b4db351-tmp-dir\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a35c68-067f-4fac-82f0-a7ef888b21fb-iptables-alerter-script\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndvj\" (UniqueName: \"kubernetes.io/projected/34a35c68-067f-4fac-82f0-a7ef888b21fb-kube-api-access-6ndvj\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-k8s-cni-cncf-io\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-kubernetes\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-conf\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-run\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.571619 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8ec8714-849f-4245-b41e-cb84f8a4996f-serviceca\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-netns\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-lib-modules\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571661    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-etc-kubernetes\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-var-lib-kubelet\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571713    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-var-lib-kubelet\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-kubelet-config\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-system-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571746    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-kubelet-config\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-netns\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-netns\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-multus\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-system-cni-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-daemon-config\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8ec8714-849f-4245-b41e-cb84f8a4996f-host\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571877    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-script-lib\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.572227 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571884    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cni-binary-copy\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-kubernetes\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a35c68-067f-4fac-82f0-a7ef888b21fb-host-slash\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571943    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-systemd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571985    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-multus\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.571989    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-conf\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-system-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572243    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-k8s-cni-cncf-io\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-conf-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-socket-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-device-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572337    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-dbus\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-systemd-units\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572388    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-run\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.572981 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-systemd-units\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572418    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a35c68-067f-4fac-82f0-a7ef888b21fb-host-slash\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-config\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-env-overrides\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-host\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572468    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-daemon-config\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cnibin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2d4\" (UniqueName: \"kubernetes.io/projected/d8ec8714-849f-4245-b41e-cb84f8a4996f-kube-api-access-mz2d4\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-ovn\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a35c68-067f-4fac-82f0-a7ef888b21fb-iptables-alerter-script\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572651    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-sys\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572709    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-conf-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572812    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-socket-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572836    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-device-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572848    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8ec8714-849f-4245-b41e-cb84f8a4996f-serviceca\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572854    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-netns\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.573751 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-tuned\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-config\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572923    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cnibin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572926    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-lib-modules\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572948    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-env-overrides\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572974    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-host\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.572978    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-ovn\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573045    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cnibin\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-kubelet\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573159    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-kubelet\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwczd\" (UniqueName: \"kubernetes.io/projected/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-kube-api-access-bwczd\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-log-socket\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573224    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cnibin\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-registration-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvvm\" (UniqueName: \"kubernetes.io/projected/78779265-8cc2-4286-8ad2-3aed34f341dd-kube-api-access-bgvvm\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.574549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-log-socket\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-node-log\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-bin\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.573339    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-netd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573354    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-registration-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dbg\" (UniqueName: \"kubernetes.io/projected/de90b397-c8f3-4b8e-b89a-822b7bc465bd-kube-api-access-79dbg\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573401    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-node-log\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.573419    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret podName:64d157fc-a912-44a3-9359-23691282dda2 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:09.073381838 +0000 UTC m=+2.270648617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret") pod "global-pull-secret-syncer-9m4h8" (UID: "64d157fc-a912-44a3-9359-23691282dda2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52674954-19d1-447d-ade5-d8655b4db351-hosts-file\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-kubelet\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-modprobe-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-tmp\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-bin\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573569    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-netd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-agent-certs\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:08.575162 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573630    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-kubelet\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573665    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52674954-19d1-447d-ade5-d8655b4db351-hosts-file\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573671    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-modprobe-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-slash\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-var-lib-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573757    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-slash\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrbx\" (UniqueName: \"kubernetes.io/projected/481d442d-4aa0-40bb-b723-ed569f0386ca-kube-api-access-sbrbx\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-systemd\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.573789    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-bin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.573852    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:09.073834165 +0000 UTC m=+2.271100941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573862    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-bin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573794    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-var-lib-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.575708 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-hostroot\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-konnectivity-ca\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-systemd\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573970    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-hostroot\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.573998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p75t\" (UniqueName: \"kubernetes.io/projected/52674954-19d1-447d-ade5-d8655b4db351-kube-api-access-5p75t\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-multus-certs\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-etc-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/481d442d-4aa0-40bb-b723-ed569f0386ca-ovn-node-metrics-cert\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysconfig\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysconfig\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574305    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-multus-certs\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-etc-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574407    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574510    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.576573 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.574762    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.577396 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.575676    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.577396 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.575989    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-konnectivity-ca\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:08.577396 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.576535    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-tuned\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.577396 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.576563    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-tmp\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.577396 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.576805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-agent-certs\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:08.578076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.578057    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/481d442d-4aa0-40bb-b723-ed569f0386ca-ovn-node-metrics-cert\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.627133 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.627097    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:08.627133 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.627125    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:08.627133 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.627139    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jxvkj for pod openshift-network-diagnostics/network-check-target-przls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:08.627360 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:08.627200    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj podName:c0b00385-648c-4505-a9a8-75070661432f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:09.127183344 +0000 UTC m=+2.324450100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jxvkj" (UniqueName: "kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj") pod "network-check-target-przls" (UID: "c0b00385-648c-4505-a9a8-75070661432f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:08.630621 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.630128    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp25t\" (UniqueName: \"kubernetes.io/projected/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-kube-api-access-dp25t\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.630621 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.630541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndvj\" (UniqueName: \"kubernetes.io/projected/34a35c68-067f-4fac-82f0-a7ef888b21fb-kube-api-access-6ndvj\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.631170 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.631132    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwczd\" (UniqueName: \"kubernetes.io/projected/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-kube-api-access-bwczd\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.631494 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.631443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dbg\" (UniqueName: \"kubernetes.io/projected/de90b397-c8f3-4b8e-b89a-822b7bc465bd-kube-api-access-79dbg\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:08.631827 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.631795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrvd6\" (UniqueName: \"kubernetes.io/projected/538a6445-5d03-43a2-a41c-0c8e89a53553-kube-api-access-hrvd6\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:08.631960 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.631940    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p75t\" (UniqueName: \"kubernetes.io/projected/52674954-19d1-447d-ade5-d8655b4db351-kube-api-access-5p75t\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.632298 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.632259    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrbx\" (UniqueName: \"kubernetes.io/projected/481d442d-4aa0-40bb-b723-ed569f0386ca-kube-api-access-sbrbx\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:08.632587 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.632568    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvvm\" (UniqueName: \"kubernetes.io/projected/78779265-8cc2-4286-8ad2-3aed34f341dd-kube-api-access-bgvvm\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:08.633187 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.633171    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2d4\" (UniqueName: \"kubernetes.io/projected/d8ec8714-849f-4245-b41e-cb84f8a4996f-kube-api-access-mz2d4\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:08.746107 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.746071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:08.767861 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.767831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:08.788228 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.788202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-586s4"
Nov 02 17:22:08.804902 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.804868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:08.822859 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:08.822827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:10.036352 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.036329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:10.041891 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.041850    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a902dc8a4f9241e56ab9949e623f113.slice/crio-1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8 WatchSource:0}: Error finding container 1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8: Status 404 returned error can't find the container with id 1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8
Nov 02 17:22:10.042159 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.042103    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5013ef74faf1be704b590476b00f1a.slice/crio-5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9 WatchSource:0}: Error finding container 5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9: Status 404 returned error can't find the container with id 5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9
Nov 02 17:22:10.042961 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.042935    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c62335d_01f0_4c4d_a2cc_4a9c3ddc4bd5.slice/crio-4ef688615f21b2e3464110833e350512b0db70dd90dbb041062bbadab3aa25ad WatchSource:0}: Error finding container 4ef688615f21b2e3464110833e350512b0db70dd90dbb041062bbadab3aa25ad: Status 404 returned error can't find the container with id 4ef688615f21b2e3464110833e350512b0db70dd90dbb041062bbadab3aa25ad
Nov 02 17:22:10.046084 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.046067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:10.047765 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.047743    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78779265_8cc2_4286_8ad2_3aed34f341dd.slice/crio-9e096c81b5a8a3e2a43d3433df16c8b6ce1bdd076f4f3ad43a3dddbb09c9c714 WatchSource:0}: Error finding container 9e096c81b5a8a3e2a43d3433df16c8b6ce1bdd076f4f3ad43a3dddbb09c9c714: Status 404 returned error can't find the container with id 9e096c81b5a8a3e2a43d3433df16c8b6ce1bdd076f4f3ad43a3dddbb09c9c714
Nov 02 17:22:10.048668 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.048655    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:22:10.058137 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.058091    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ec8714_849f_4245_b41e_cb84f8a4996f.slice/crio-46b28861eecd2b516f7ff22ba695762cf36c736aac395ec261c0467ef7c4191b WatchSource:0}: Error finding container 46b28861eecd2b516f7ff22ba695762cf36c736aac395ec261c0467ef7c4191b: Status 404 returned error can't find the container with id 46b28861eecd2b516f7ff22ba695762cf36c736aac395ec261c0467ef7c4191b
Nov 02 17:22:10.081970 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.081899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:10.084002 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.083977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:10.092171 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.092137    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481d442d_4aa0_40bb_b723_ed569f0386ca.slice/crio-fb8780c28b5b1810eb66007fb0ae9ca771493aeb3ac75963abaf4f7acc702202 WatchSource:0}: Error finding container fb8780c28b5b1810eb66007fb0ae9ca771493aeb3ac75963abaf4f7acc702202: Status 404 returned error can't find the container with id fb8780c28b5b1810eb66007fb0ae9ca771493aeb3ac75963abaf4f7acc702202
Nov 02 17:22:10.095253 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.095233    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde90b397_c8f3_4b8e_b89a_822b7bc465bd.slice/crio-b4f24b1f9f701bacad06ad04a0ada80a8048f176b43721328105824ef67e7a71 WatchSource:0}: Error finding container b4f24b1f9f701bacad06ad04a0ada80a8048f176b43721328105824ef67e7a71: Status 404 returned error can't find the container with id b4f24b1f9f701bacad06ad04a0ada80a8048f176b43721328105824ef67e7a71
Nov 02 17:22:10.269750 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.269710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:10.269750 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.269747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:10.269949 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.269844    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:10.269949 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.269846    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:10.269949 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.269892    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:11.269880322 +0000 UTC m=+3.276291743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:10.269949 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.269904    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret podName:64d157fc-a912-44a3-9359-23691282dda2 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:11.269898499 +0000 UTC m=+3.276309920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret") pod "global-pull-secret-syncer-9m4h8" (UID: "64d157fc-a912-44a3-9359-23691282dda2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:10.334893 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.334809    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:10.370267 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.370238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:10.370413 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.370385    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:10.370413 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.370399    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:10.370413 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.370408    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jxvkj for pod openshift-network-diagnostics/network-check-target-przls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:10.370528 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:10.370455    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj podName:c0b00385-648c-4505-a9a8-75070661432f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:11.370444143 +0000 UTC m=+3.376855566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jxvkj" (UniqueName: "kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj") pod "network-check-target-przls" (UID: "c0b00385-648c-4505-a9a8-75070661432f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:10.440175 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.440149    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:10.497600 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.497545    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff" event={"ID":"78779265-8cc2-4286-8ad2-3aed34f341dd","Type":"ContainerStarted","Data":"9e096c81b5a8a3e2a43d3433df16c8b6ce1bdd076f4f3ad43a3dddbb09c9c714"}
Nov 02 17:22:10.498564 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.498540    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"7a902dc8a4f9241e56ab9949e623f113","Type":"ContainerStarted","Data":"1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8"}
Nov 02 17:22:10.499481 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.499447    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"8b5013ef74faf1be704b590476b00f1a","Type":"ContainerStarted","Data":"5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9"}
Nov 02 17:22:10.500504 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.500485    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7rhm" event={"ID":"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5","Type":"ContainerStarted","Data":"4ef688615f21b2e3464110833e350512b0db70dd90dbb041062bbadab3aa25ad"}
Nov 02 17:22:10.501499 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.501474    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q26xm" event={"ID":"de90b397-c8f3-4b8e-b89a-822b7bc465bd","Type":"ContainerStarted","Data":"b4f24b1f9f701bacad06ad04a0ada80a8048f176b43721328105824ef67e7a71"}
Nov 02 17:22:10.502650 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.502633    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"fb8780c28b5b1810eb66007fb0ae9ca771493aeb3ac75963abaf4f7acc702202"}
Nov 02 17:22:10.503628 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.503612    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gmfdg" event={"ID":"d8ec8714-849f-4245-b41e-cb84f8a4996f","Type":"ContainerStarted","Data":"46b28861eecd2b516f7ff22ba695762cf36c736aac395ec261c0467ef7c4191b"}
Nov 02 17:22:10.727180 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.727132    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:09 +0000 UTC" deadline="2028-06-01 01:32:05.147259033 +0000 UTC"
Nov 02 17:22:10.727180 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.727173    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13832h9m54.420088197s"
Nov 02 17:22:10.779560 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.779535    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45ecbf9_1078_4ba0_bb04_0684bc7aadd3.slice/crio-7effc1e998a71350420a8aeef9b373b0ddfee241c408a1d4b2a890d2ae18ff0d WatchSource:0}: Error finding container 7effc1e998a71350420a8aeef9b373b0ddfee241c408a1d4b2a890d2ae18ff0d: Status 404 returned error can't find the container with id 7effc1e998a71350420a8aeef9b373b0ddfee241c408a1d4b2a890d2ae18ff0d
Nov 02 17:22:10.829240 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:10.829209    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:10.833218 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.833192    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b519ef8_e177_4fb8_9cb0_0e3ee48b2be5.slice/crio-4a3bac5553253fb4f68decc088f6ade27aadfe0726774073c9bc74ea8008eed5 WatchSource:0}: Error finding container 4a3bac5553253fb4f68decc088f6ade27aadfe0726774073c9bc74ea8008eed5: Status 404 returned error can't find the container with id 4a3bac5553253fb4f68decc088f6ade27aadfe0726774073c9bc74ea8008eed5
Nov 02 17:22:10.922318 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:10.922293    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a35c68_067f_4fac_82f0_a7ef888b21fb.slice/crio-4906b6dee53066d5e6f1336ddc39c5dd9fb95ce08587ff4af6fc36005115645f WatchSource:0}: Error finding container 4906b6dee53066d5e6f1336ddc39c5dd9fb95ce08587ff4af6fc36005115645f: Status 404 returned error can't find the container with id 4906b6dee53066d5e6f1336ddc39c5dd9fb95ce08587ff4af6fc36005115645f
Nov 02 17:22:11.279894 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.279467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:11.279894 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.279531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:11.279894 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.279678    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:11.279894 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.279739    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:13.279721864 +0000 UTC m=+5.286133329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:11.279894 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.279800    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:11.279894 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.279830    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret podName:64d157fc-a912-44a3-9359-23691282dda2 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:13.279820414 +0000 UTC m=+5.286231839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret") pod "global-pull-secret-syncer-9m4h8" (UID: "64d157fc-a912-44a3-9359-23691282dda2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:11.380361 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.380323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:11.380808 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.380512    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:11.380808 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.380538    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:11.380808 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.380550    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jxvkj for pod openshift-network-diagnostics/network-check-target-przls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:11.380808 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.380608    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj podName:c0b00385-648c-4505-a9a8-75070661432f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:13.380591804 +0000 UTC m=+5.387003228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jxvkj" (UniqueName: "kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj") pod "network-check-target-przls" (UID: "c0b00385-648c-4505-a9a8-75070661432f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:11.494432 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.494398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:11.494647 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.494533    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:11.494647 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.494612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:11.494784 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.494681    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:11.494784 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.494743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:11.494903 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:11.494819    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:11.517322 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.517285    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mj6cq" event={"ID":"34a35c68-067f-4fac-82f0-a7ef888b21fb","Type":"ContainerStarted","Data":"4906b6dee53066d5e6f1336ddc39c5dd9fb95ce08587ff4af6fc36005115645f"}
Nov 02 17:22:11.521397 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.521365    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-586s4" event={"ID":"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5","Type":"ContainerStarted","Data":"4a3bac5553253fb4f68decc088f6ade27aadfe0726774073c9bc74ea8008eed5"}
Nov 02 17:22:11.524058 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.524029    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerStarted","Data":"7effc1e998a71350420a8aeef9b373b0ddfee241c408a1d4b2a890d2ae18ff0d"}
Nov 02 17:22:11.525668 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.525647    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zfl47" event={"ID":"52674954-19d1-447d-ade5-d8655b4db351","Type":"ContainerStarted","Data":"14a17a4a79ac75507316c26c5d5444dd64134161c541975c36c57edc089db83a"}
Nov 02 17:22:11.728337 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.728269    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:09 +0000 UTC" deadline="2028-07-02 04:51:49.533538997 +0000 UTC"
Nov 02 17:22:11.728337 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:11.728336    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14579h29m37.805207123s"
Nov 02 17:22:13.295806 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:13.295769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:13.296320 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:13.295829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:13.296320 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.295990    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:13.296320 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.296048    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:17.296029798 +0000 UTC m=+9.302441243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:13.296320 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.296152    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:13.296320 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.296187    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret podName:64d157fc-a912-44a3-9359-23691282dda2 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:17.296175309 +0000 UTC m=+9.302586737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret") pod "global-pull-secret-syncer-9m4h8" (UID: "64d157fc-a912-44a3-9359-23691282dda2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:13.396548 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:13.396509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:13.396742 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.396686    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:13.396742 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.396710    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:13.396742 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.396724    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jxvkj for pod openshift-network-diagnostics/network-check-target-przls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:13.396906 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.396805    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj podName:c0b00385-648c-4505-a9a8-75070661432f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:17.396766136 +0000 UTC m=+9.403177561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jxvkj" (UniqueName: "kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj") pod "network-check-target-przls" (UID: "c0b00385-648c-4505-a9a8-75070661432f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:13.494436 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:13.494320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:13.494436 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:13.494346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:13.494681 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.494482    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:13.494681 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.494583    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:13.494681 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:13.494636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:13.494830 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:13.494800    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:15.493829 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.493795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:15.494326 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:15.493938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:15.494326 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.494031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:15.494326 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:15.494106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:15.494326 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.494205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:15.494326 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:15.494308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:15.551744 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.551705    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7rhm" event={"ID":"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5","Type":"ContainerStarted","Data":"509fafc28513fd49f9c45834a0b204c3b1bd5169b2db4ca0c4167b445c18a0eb"}
Nov 02 17:22:15.558115 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.558078    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="2084fc8f05cb7e14b617068f5e616daba23457ffdab4ca45d30acfe3e377de54" exitCode=0
Nov 02 17:22:15.558288 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.558195    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerDied","Data":"2084fc8f05cb7e14b617068f5e616daba23457ffdab4ca45d30acfe3e377de54"}
Nov 02 17:22:15.563355 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.563325    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"7a902dc8a4f9241e56ab9949e623f113","Type":"ContainerStarted","Data":"38d40e12a7ebee8ef6e7d765747294d39d48aba25d9fe9111f9fa1a17dcaa7fa"}
Nov 02 17:22:15.566927 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.566903    2578 generic.go:356] "Generic (PLEG): container finished" podID="8b5013ef74faf1be704b590476b00f1a" containerID="926e43b948b600892e01bfe80eacda89c7218c459283ad0bf181aa5a89d1a92b" exitCode=0
Nov 02 17:22:15.567001 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.566945    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"8b5013ef74faf1be704b590476b00f1a","Type":"ContainerDied","Data":"926e43b948b600892e01bfe80eacda89c7218c459283ad0bf181aa5a89d1a92b"}
Nov 02 17:22:15.798615 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:15.797661    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m7rhm" podStartSLOduration=3.427252377 podStartE2EDuration="7.797641072s" totalImagesPullingTime="4.370388695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:15.670091428 +0000 UTC m=+7.676502869" watchObservedRunningTime="2026-11-02 17:22:15.797641072 +0000 UTC m=+7.804052516"
Nov 02 17:22:16.470591 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:16.470552    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:16.471421 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:16.471394    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:16.574516 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:16.574476    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gmfdg" event={"ID":"d8ec8714-849f-4245-b41e-cb84f8a4996f","Type":"ContainerStarted","Data":"1ba288ce4da2271fc3c7742b79ad31f0407b1433d38ef503ecdb2d546b323369"}
Nov 02 17:22:16.580822 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:16.580789    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"8b5013ef74faf1be704b590476b00f1a","Type":"ContainerStarted","Data":"36454a7414909d1de154461df347ffcec1ad8c6b186d05678d137d146238191e"}
Nov 02 17:22:16.588209 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:16.588156    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal" podStartSLOduration=8.58813851 podStartE2EDuration="8.58813851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:16.024434972 +0000 UTC m=+8.030846415" watchObservedRunningTime="2026-11-02 17:22:16.58813851 +0000 UTC m=+8.594549951"
Nov 02 17:22:16.825669 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:16.825622    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal" podStartSLOduration=8.825606142 podStartE2EDuration="8.825606142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:16.703153282 +0000 UTC m=+8.709564727" watchObservedRunningTime="2026-11-02 17:22:16.825606142 +0000 UTC m=+8.832017584"
Nov 02 17:22:17.337588 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.337155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:17.337588 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.337211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:17.337588 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.337377    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:17.337588 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.337443    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:25.3374227 +0000 UTC m=+17.343834136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:17.337588 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.337508    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:17.337588 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.337540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret podName:64d157fc-a912-44a3-9359-23691282dda2 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:25.337529516 +0000 UTC m=+17.343940952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret") pod "global-pull-secret-syncer-9m4h8" (UID: "64d157fc-a912-44a3-9359-23691282dda2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:17.438640 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.438593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:17.439323 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.439291    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:17.439462 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.439360    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:17.439462 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.439382    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jxvkj for pod openshift-network-diagnostics/network-check-target-przls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:17.439583 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.439520    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj podName:c0b00385-648c-4505-a9a8-75070661432f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:25.439474803 +0000 UTC m=+17.445886242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jxvkj" (UniqueName: "kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj") pod "network-check-target-przls" (UID: "c0b00385-648c-4505-a9a8-75070661432f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:17.494542 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.494504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:17.494542 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.494536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:17.494542 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.494504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:17.494782 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.494647    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:17.494782 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.494688    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:17.494782 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:17.494731    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:17.585435 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.585397    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff" event={"ID":"78779265-8cc2-4286-8ad2-3aed34f341dd","Type":"ContainerStarted","Data":"90590bbac72b4f2213f6701011c84d55606761586e913d325ac0316e496f6553"}
Nov 02 17:22:17.585921 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:17.585504    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:22:18.600819 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:18.599736    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-586s4" event={"ID":"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5","Type":"ContainerStarted","Data":"532baa92dfd0bb05236d3e2877ad2cbde684eaa9d0c3f1d668168c5fdc410f0e"}
Nov 02 17:22:18.714486 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:18.714380    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gmfdg" podStartSLOduration=5.023018876 podStartE2EDuration="10.714361151s" totalImagesPullingTime="5.691342275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:16.826573982 +0000 UTC m=+8.832985425" watchObservedRunningTime="2026-11-02 17:22:18.714361151 +0000 UTC m=+10.720772593"
Nov 02 17:22:18.715011 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:18.714982    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-586s4" podStartSLOduration=3.096085912 podStartE2EDuration="10.714975623s" totalImagesPullingTime="7.618889711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:18.714200451 +0000 UTC m=+10.720611896" watchObservedRunningTime="2026-11-02 17:22:18.714975623 +0000 UTC m=+10.721387067"
Nov 02 17:22:18.875369 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:18.875307    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:22:19.433022 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.432905    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:22:18.87533048Z","UUID":"ad4c2c15-74ca-4213-b44e-49befe41bb44","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:22:19.434919 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.434894    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:22:19.434919 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.434925    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:22:19.494439 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.494409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:19.494582 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.494451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:19.494582 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.494415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:19.494582 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:19.494549    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:19.494755 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:19.494639    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:19.494755 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:19.494718    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:19.603635 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.603595    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zfl47" event={"ID":"52674954-19d1-447d-ade5-d8655b4db351","Type":"ContainerStarted","Data":"34882b1b99a34d5ff54fda9599d0fc8d944285e1ea48eaf4243128c308d38920"}
Nov 02 17:22:19.606358 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:19.606326    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff" event={"ID":"78779265-8cc2-4286-8ad2-3aed34f341dd","Type":"ContainerStarted","Data":"0a2a9137a7cc6c478794d8235448e7287cc9d45addcc2109b8e18a387683b4c6"}
Nov 02 17:22:20.610551 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:20.610442    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="6873eeb1df5eac7e0d251db4bfc9a3782d2fdbb9ae19b4ed1c519e6d19a5eff8" exitCode=0
Nov 02 17:22:20.610551 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:20.610482    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerDied","Data":"6873eeb1df5eac7e0d251db4bfc9a3782d2fdbb9ae19b4ed1c519e6d19a5eff8"}
Nov 02 17:22:20.613920 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:20.613886    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff" event={"ID":"78779265-8cc2-4286-8ad2-3aed34f341dd","Type":"ContainerStarted","Data":"bf174f0f8e36d45860f7ca6e11b8e1bc5792b9abd99d4de20b2e389e5e13d47c"}
Nov 02 17:22:20.726204 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:20.726149    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zfl47" podStartSLOduration=4.102826404 podStartE2EDuration="12.726134605s" totalImagesPullingTime="8.623308201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:19.717488529 +0000 UTC m=+11.723899973" watchObservedRunningTime="2026-11-02 17:22:20.726134605 +0000 UTC m=+12.732546041"
Nov 02 17:22:20.839229 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:20.839168    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff" podStartSLOduration=2.859293502 podStartE2EDuration="12.839149055s" totalImagesPullingTime="9.979855553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:20.838726496 +0000 UTC m=+12.845137942" watchObservedRunningTime="2026-11-02 17:22:20.839149055 +0000 UTC m=+12.845560499"
Nov 02 17:22:21.493827 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:21.493793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:21.493997 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:21.493865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:21.493997 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:21.493956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:21.493997 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:21.493971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:21.494142 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:21.494080    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:21.494202 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:21.494182    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:21.616963 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:21.616927    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q26xm" event={"ID":"de90b397-c8f3-4b8e-b89a-822b7bc465bd","Type":"ContainerStarted","Data":"54a704e29dd14068ee069dfc16ebd928419550f42d6707b686afc4b2cae6d932"}
Nov 02 17:22:21.618726 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:21.618532    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mj6cq" event={"ID":"34a35c68-067f-4fac-82f0-a7ef888b21fb","Type":"ContainerStarted","Data":"af50206efdc4a556b24b9da303fabecc95d030be105af63e0abfabe4e6b750c1"}
Nov 02 17:22:21.731813 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:21.731753    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q26xm" podStartSLOduration=2.955846533 podStartE2EDuration="13.731735512s" totalImagesPullingTime="10.775888979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:21.731581502 +0000 UTC m=+13.737992946" watchObservedRunningTime="2026-11-02 17:22:21.731735512 +0000 UTC m=+13.738146957"
Nov 02 17:22:21.844752 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:21.844622    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mj6cq" podStartSLOduration=5.6384712310000005 podStartE2EDuration="13.844600364s" totalImagesPullingTime="8.206129133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:21.844180847 +0000 UTC m=+13.850592288" watchObservedRunningTime="2026-11-02 17:22:21.844600364 +0000 UTC m=+13.851011809"
Nov 02 17:22:22.623515 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:22.623465    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="04b8357358cbc9aae24b2303a1404b2b776f2178786a2c47d3d794b620908c3b" exitCode=0
Nov 02 17:22:22.623931 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:22.623563    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerDied","Data":"04b8357358cbc9aae24b2303a1404b2b776f2178786a2c47d3d794b620908c3b"}
Nov 02 17:22:23.494427 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:23.494371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:23.494427 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:23.494404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:23.494706 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:23.494417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:23.494706 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:23.494518    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:23.494706 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:23.494594    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:23.494706 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:23.494676    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:24.630635 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:24.630594    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="401db22b1ff1a9752ed9dffa3d71154df0e9682213cd5caf2c0c17a76ff7eea9" exitCode=0
Nov 02 17:22:24.631031 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:24.630652    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerDied","Data":"401db22b1ff1a9752ed9dffa3d71154df0e9682213cd5caf2c0c17a76ff7eea9"}
Nov 02 17:22:25.397620 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.397440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:25.397709 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.397582    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:25.397754 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.397725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:25.397787 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.397765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:41.397743099 +0000 UTC m=+33.404154545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:25.397831 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.397820    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:25.397884 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.397874    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret podName:64d157fc-a912-44a3-9359-23691282dda2 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:41.397858186 +0000 UTC m=+33.404269611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret") pod "global-pull-secret-syncer-9m4h8" (UID: "64d157fc-a912-44a3-9359-23691282dda2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:25.494319 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.494297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:25.494452 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.494335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:25.494452 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.494437    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:25.494527 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.494470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:25.494596 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.494567    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:25.494661 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.494639    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:25.498951 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.498935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:25.499085 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.499072    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:25.499137 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.499091    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:25.499137 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.499099    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jxvkj for pod openshift-network-diagnostics/network-check-target-przls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:25.499196 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:25.499157    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj podName:c0b00385-648c-4505-a9a8-75070661432f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:41.499142632 +0000 UTC m=+33.505554058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jxvkj" (UniqueName: "kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj") pod "network-check-target-przls" (UID: "c0b00385-648c-4505-a9a8-75070661432f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:25.635636 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.635611    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"979eec6966b8546fdc432359dbedc05cb4ac56ad753e490c3a11a75e8a2f45dd"}
Nov 02 17:22:25.636097 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.635682    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"ebb552e8b356dae9385b309b4b484d8c53cb665a5300526d1c63646bc92b95da"}
Nov 02 17:22:25.636097 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.635711    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"e6913a7ffc884702da87d36daa089bfaef8a1f25084b293aa476660ef93f0dfe"}
Nov 02 17:22:25.636097 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.635738    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"be2450c087f25f42977357a88d46ef6b62e716afc6694fd61e0ef3e79a9b7ebc"}
Nov 02 17:22:25.636097 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:25.635762    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"fce8f02a8854bc6741580ace11f68131924fc0d8b5224dee55feaaed7b6a3fe3"}
Nov 02 17:22:26.641441 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:26.641407    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"243c22a59799e9165c56d1c4232f69eb3bef2a0604c6c8b90a65827a849d2cb0"}
Nov 02 17:22:27.494293 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:27.494258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:27.494293 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:27.494295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:27.494524 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:27.494360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:27.494524 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:27.494407    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:27.494524 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:27.494452    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:27.494524 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:27.494504    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:27.645589 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:27.645560    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="090f004b972e77cdc9c095d41a41852a7edb0de5e613d932c83d99d8cd074ef3" exitCode=0
Nov 02 17:22:27.645589 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:27.645593    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerDied","Data":"090f004b972e77cdc9c095d41a41852a7edb0de5e613d932c83d99d8cd074ef3"}
Nov 02 17:22:28.650453 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:28.650417    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"5758628aa6a09f1efba3e2ff24e96fa1365fc3fcf7dc24320e195ad82f727f5a"}
Nov 02 17:22:28.653036 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:28.653010    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="0491ceacad4e56a5e30e7411e0ed52cb3364817644ae8bbc2426dfb5b6f2eedc" exitCode=0
Nov 02 17:22:28.653165 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:28.653036    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerDied","Data":"0491ceacad4e56a5e30e7411e0ed52cb3364817644ae8bbc2426dfb5b6f2eedc"}
Nov 02 17:22:29.493844 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:29.493805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:29.494021 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:29.493826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:29.494021 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:29.493927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:29.494021 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:29.493928    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:29.494021 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:29.493994    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:29.494182 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:29.494071    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:29.657690 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:29.657659    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" event={"ID":"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3","Type":"ContainerStarted","Data":"f10bfd7d3e1d30bdf6bc5a92f5e477886b5f47d952c5e1fa7485e9fb22b7bd47"}
Nov 02 17:22:29.777053 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:29.776952    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g7sgh" podStartSLOduration=9.332288454 podStartE2EDuration="21.776940495s" totalImagesPullingTime="12.444652041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:29.776639485 +0000 UTC m=+21.783050929" watchObservedRunningTime="2026-11-02 17:22:29.776940495 +0000 UTC m=+21.783351937"
Nov 02 17:22:30.663681 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:30.663477    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" event={"ID":"481d442d-4aa0-40bb-b723-ed569f0386ca","Type":"ContainerStarted","Data":"3b5677601450b713abcdff651e30679c88af88ceb6a790094620c194a344cf59"}
Nov 02 17:22:30.937102 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:30.937009    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-455n9" podStartSLOduration=7.837234714 podStartE2EDuration="22.936994127s" totalImagesPullingTime="15.099759413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:30.885518359 +0000 UTC m=+22.891929824" watchObservedRunningTime="2026-11-02 17:22:30.936994127 +0000 UTC m=+22.943405568"
Nov 02 17:22:31.494025 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.493988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:31.494241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.493999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:31.494241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:31.494098    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:31.494241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:31.494171    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:31.494241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.494012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:31.494241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:31.494233    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:31.666056 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.666027    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:31.666734 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.666102    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:31.666734 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.666163    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:31.682331 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.682306    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:31.682531 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:31.682521    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:32.605512 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:32.605478    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-przls"]
Nov 02 17:22:32.605715 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:32.605608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:32.605715 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:32.605696    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:32.609622 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:32.609595    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7btw"]
Nov 02 17:22:32.609758 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:32.609654    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9m4h8"]
Nov 02 17:22:32.609758 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:32.609734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:32.609857 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:32.609823    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:32.609915 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:32.609897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:32.609997 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:32.609980    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:34.494355 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:34.494313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:34.494844 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:34.494314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:34.494844 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:34.494468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:34.494844 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:34.494470    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:34.494844 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:34.494536    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:34.494844 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:34.494622    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:36.071050 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:36.071015    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:36.071552 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:36.071224    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:22:36.071773 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:36.071750    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:36.495579 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:36.495544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:36.495779 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:36.495605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:36.495779 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:36.495618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:36.495779 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:36.495707    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9m4h8" podUID="64d157fc-a912-44a3-9359-23691282dda2"
Nov 02 17:22:36.495941 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:36.495773    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-przls" podUID="c0b00385-648c-4505-a9a8-75070661432f"
Nov 02 17:22:36.495941 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:36.495829    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7btw" podUID="538a6445-5d03-43a2-a41c-0c8e89a53553"
Nov 02 17:22:38.250381 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.250350    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeReady"
Nov 02 17:22:38.250757 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.250515    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 17:22:38.364922 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.364885    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tbpf9"]
Nov 02 17:22:38.367829 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.367775    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-czpdz"]
Nov 02 17:22:38.367988 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.367877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tbpf9"
Nov 02 17:22:38.369452 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.369428    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gkwvc"]
Nov 02 17:22:38.369565 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.369520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-czpdz"
Nov 02 17:22:38.370867 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.370851    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76ff76747b-h7b52"]
Nov 02 17:22:38.370979 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.370965    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gkwvc"
Nov 02 17:22:38.372169 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.372142    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"]
Nov 02 17:22:38.372323 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.372309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.373431 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.373415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:38.376846 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.376826    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"]
Nov 02 17:22:38.378541 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.378522    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"]
Nov 02 17:22:38.378618 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.378591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.380011 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.379995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.409650 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.409630    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67778b997c-255bm"]
Nov 02 17:22:38.411239 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.411218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.415967 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.415945    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-5hwls"]
Nov 02 17:22:38.417863 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.417844    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d9d94667c-xzxd4"]
Nov 02 17:22:38.417967 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.417954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.419622 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.419603    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"]
Nov 02 17:22:38.419748 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.419733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.421279 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.421261    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"]
Nov 02 17:22:38.421374 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.421364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.421798 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.421766    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.421882 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.421837    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.422993 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.422976    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"]
Nov 02 17:22:38.423075 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.423060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.425390 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.425372    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"]
Nov 02 17:22:38.425484 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.425468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"
Nov 02 17:22:38.428113 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.428094    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"]
Nov 02 17:22:38.428289 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.428272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.429938 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.429922    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 17:22:38.430396 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.430381    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rn2gc"]
Nov 02 17:22:38.430527 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.430510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"
Nov 02 17:22:38.431866 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.431849    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8v2j6"]
Nov 02 17:22:38.431929 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.431920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.433439 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.433424    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-czpdz"]
Nov 02 17:22:38.433526 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.433455    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gkwvc"]
Nov 02 17:22:38.433526 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.433474    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tbpf9"]
Nov 02 17:22:38.433624 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.433531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.461228 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.461203    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76ff76747b-h7b52"]
Nov 02 17:22:38.478254 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.478224    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"]
Nov 02 17:22:38.487263 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-config\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.487415 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.487415 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:38.487415 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.487415 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.487631 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e170d8-e67b-4972-9e92-28b033c07336-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.487631 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-tmp\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.487631 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.487631 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsr5f\" (UniqueName: \"kubernetes.io/projected/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc-kube-api-access-bsr5f\") pod \"kas-connection-checker-685895c7c9-czpdz\" (UID: \"c8f53a9e-fe98-4056-8d6b-29bd6f1912cc\") " pod="kube-system/kas-connection-checker-685895c7c9-czpdz"
Nov 02 17:22:38.487631 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487373    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"]
Nov 02 17:22:38.487631 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wxdc\" (UniqueName: \"kubernetes.io/projected/cd56f1da-de64-4541-9c26-e5ba4b221dfa-kube-api-access-9wxdc\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslwl\" (UniqueName: \"kubernetes.io/projected/44e170d8-e67b-4972-9e92-28b033c07336-kube-api-access-vslwl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e170d8-e67b-4972-9e92-28b033c07336-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-tmp-dir\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-config\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.487942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.488000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.488027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqw5\" (UniqueName: \"kubernetes.io/projected/c7260359-2132-4442-8388-5cd2455b5700-kube-api-access-mvqw5\") pod \"network-check-source-6f78d59f99-988dm\" (UID: \"c7260359-2132-4442-8388-5cd2455b5700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.488100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs858\" (UniqueName: \"kubernetes.io/projected/8a562913-9258-41b3-960e-c9dc07e2637c-kube-api-access-xs858\") pod \"kas-connection-checker-685895c7c9-gkwvc\" (UID: \"8a562913-9258-41b3-960e-c9dc07e2637c\") " pod="kube-system/kas-connection-checker-685895c7c9-gkwvc"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.488253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.488373    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-5hwls"]
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.488447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5998295-35aa-458b-9174-625f915c16cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-serving-cert\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.489224 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxgv\" (UniqueName: \"kubernetes.io/projected/f2fc2433-5459-47d9-a32f-511bf10bcf70-kube-api-access-5qxgv\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflzw\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-service-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8mg\" (UniqueName: \"kubernetes.io/projected/aae52ad6-e69f-4db7-8924-3927e0b9af50-kube-api-access-rm8mg\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/45fb7f7a-5ad6-4176-8405-ece930a49aca-kube-api-access-flcdx\") pod \"volume-data-source-validator-58c9f5f46d-6c5hm\" (UID: \"45fb7f7a-5ad6-4176-8405-ece930a49aca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-config-volume\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedb70cd-4f51-41ae-acfd-6c5c2044644d-serving-cert\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb5d1d52-3223-45e9-a766-848d837c029b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.490076 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aae52ad6-e69f-4db7-8924-3927e0b9af50-cert\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.489988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kr6\" (UniqueName: \"kubernetes.io/projected/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-kube-api-access-f7kr6\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqnp\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdw4\" (UniqueName: \"kubernetes.io/projected/a1767839-4632-46c7-84aa-19269b3be129-kube-api-access-lzdw4\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-stats-auth\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490360    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67778b997c-255bm"]
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmrr\" (UniqueName: \"kubernetes.io/projected/d269254c-0c37-4fad-b7e4-20b88ee2c2c1-kube-api-access-5hmrr\") pod \"kas-connection-checker-685895c7c9-tbpf9\" (UID: \"d269254c-0c37-4fad-b7e4-20b88ee2c2c1\") " pod="kube-system/kas-connection-checker-685895c7c9-tbpf9"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-trusted-ca\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-default-certificate\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskxm\" (UniqueName: \"kubernetes.io/projected/fb5d1d52-3223-45e9-a766-848d837c029b-kube-api-access-gskxm\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fc2433-5459-47d9-a32f-511bf10bcf70-serving-cert\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-snapshots\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.490917 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxz6p\" (UniqueName: \"kubernetes.io/projected/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-kube-api-access-rxz6p\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.491682 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.491682 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.490796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjmg\" (UniqueName: \"kubernetes.io/projected/eedb70cd-4f51-41ae-acfd-6c5c2044644d-kube-api-access-qtjmg\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.491682 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.491167    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"]
Nov 02 17:22:38.492871 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.492846    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"]
Nov 02 17:22:38.493983 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.493969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:38.494034 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.493983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:38.494174 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.494163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:38.523004 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.522968    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.523004 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.522968    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 17:22:38.523304 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523287    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.523385 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523300    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.523385 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523351    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.523385 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523366    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523391    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523402    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523409    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523471    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523490    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523489    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523530    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523552    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523555    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:22:38.523571 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523568    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523582    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523642    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523696    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523714    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523752    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523755    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523814    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.523824    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.524073    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.524104    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.524307 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.524188    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 17:22:38.525264 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.525245    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 17:22:38.541184 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.541156    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"]
Nov 02 17:22:38.575106 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.575072    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 17:22:38.590756 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.590729    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 17:22:38.591613 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.591250    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 17:22:38.591613 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.591455    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kr6\" (UniqueName: \"kubernetes.io/projected/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-kube-api-access-f7kr6\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592203    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqnp\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdw4\" (UniqueName: \"kubernetes.io/projected/a1767839-4632-46c7-84aa-19269b3be129-kube-api-access-lzdw4\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-stats-auth\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmrr\" (UniqueName: \"kubernetes.io/projected/d269254c-0c37-4fad-b7e4-20b88ee2c2c1-kube-api-access-5hmrr\") pod \"kas-connection-checker-685895c7c9-tbpf9\" (UID: \"d269254c-0c37-4fad-b7e4-20b88ee2c2c1\") " pod="kube-system/kas-connection-checker-685895c7c9-tbpf9"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-trusted-ca\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-default-certificate\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592712    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskxm\" (UniqueName: \"kubernetes.io/projected/fb5d1d52-3223-45e9-a766-848d837c029b-kube-api-access-gskxm\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592803    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592930    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.593006    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svxv2\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.593205    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fb2bd\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.593357    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.593414    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.593492    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.593544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.593585    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-s74tj\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594068    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-djvvn\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594104    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5vhr8\""
Nov 02 17:22:38.595215 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594110    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v649h\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594196    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7t5cr\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594286    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594308    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594346    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594516    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594658    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.592802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fc2433-5459-47d9-a32f-511bf10bcf70-serving-cert\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-snapshots\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxz6p\" (UniqueName: \"kubernetes.io/projected/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-kube-api-access-rxz6p\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.594973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjmg\" (UniqueName: \"kubernetes.io/projected/eedb70cd-4f51-41ae-acfd-6c5c2044644d-kube-api-access-qtjmg\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-config\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595086    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e170d8-e67b-4972-9e92-28b033c07336-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.595999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-tmp\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.596280 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.596035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.597067 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.596058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsr5f\" (UniqueName: \"kubernetes.io/projected/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc-kube-api-access-bsr5f\") pod \"kas-connection-checker-685895c7c9-czpdz\" (UID: \"c8f53a9e-fe98-4056-8d6b-29bd6f1912cc\") " pod="kube-system/kas-connection-checker-685895c7c9-czpdz"
Nov 02 17:22:38.597067 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.596103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wxdc\" (UniqueName: \"kubernetes.io/projected/cd56f1da-de64-4541-9c26-e5ba4b221dfa-kube-api-access-9wxdc\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.597067 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.596727    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gfr4c\""
Nov 02 17:22:38.597067 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.596996    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-trusted-ca\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.597285 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.597099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-snapshots\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.597380 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.597362    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:38.597464 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.597423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.097405668 +0000 UTC m=+31.103817094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:38.597606 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.597588    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-config\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.597665 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.597616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.597665 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.597640    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8v2j6"]
Nov 02 17:22:38.597756 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.597709    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:38.597756 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.597724    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:38.597842 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.597770    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.097752592 +0000 UTC m=+31.104164015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:38.598179 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598154    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-tmp\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.598423 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslwl\" (UniqueName: \"kubernetes.io/projected/44e170d8-e67b-4972-9e92-28b033c07336-kube-api-access-vslwl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.598500 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e170d8-e67b-4972-9e92-28b033c07336-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.598500 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.598604 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-tmp-dir\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.598604 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598519    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"]
Nov 02 17:22:38.598604 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.598604 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-config\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.598862 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.598835    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.599080 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.599220 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.599200    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:38.599290 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.599257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.09924207 +0000 UTC m=+31.105653509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:38.599290 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599269    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e170d8-e67b-4972-9e92-28b033c07336-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.599389 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599073    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-tmp-dir\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.599473 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.599544 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqw5\" (UniqueName: \"kubernetes.io/projected/c7260359-2132-4442-8388-5cd2455b5700-kube-api-access-mvqw5\") pod \"network-check-source-6f78d59f99-988dm\" (UID: \"c7260359-2132-4442-8388-5cd2455b5700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"
Nov 02 17:22:38.599544 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs858\" (UniqueName: \"kubernetes.io/projected/8a562913-9258-41b3-960e-c9dc07e2637c-kube-api-access-xs858\") pod \"kas-connection-checker-685895c7c9-gkwvc\" (UID: \"8a562913-9258-41b3-960e-c9dc07e2637c\") " pod="kube-system/kas-connection-checker-685895c7c9-gkwvc"
Nov 02 17:22:38.599648 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.599648 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5998295-35aa-458b-9174-625f915c16cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:38.599648 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-serving-cert\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.599648 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599614    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-stats-auth\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.599648 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxgv\" (UniqueName: \"kubernetes.io/projected/f2fc2433-5459-47d9-a32f-511bf10bcf70-kube-api-access-5qxgv\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflzw\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-service-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.599763    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.599776    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8mg\" (UniqueName: \"kubernetes.io/projected/aae52ad6-e69f-4db7-8924-3927e0b9af50-kube-api-access-rm8mg\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/45fb7f7a-5ad6-4176-8405-ece930a49aca-kube-api-access-flcdx\") pod \"volume-data-source-validator-58c9f5f46d-6c5hm\" (UID: \"45fb7f7a-5ad6-4176-8405-ece930a49aca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.599877 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.600349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.600349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-config-volume\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.600349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedb70cd-4f51-41ae-acfd-6c5c2044644d-serving-cert\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.600349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.599980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb5d1d52-3223-45e9-a766-848d837c029b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.600349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.600007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.600349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.600034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.600623 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.600249    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.600675 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.600364    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.100344709 +0000 UTC m=+31.106756138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:38.600675 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.600537    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-default-certificate\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.600675 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.600617    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:38.600825 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.600716    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.100700102 +0000 UTC m=+31.107111528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:38.600825 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.600737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.600924 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.600914    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:38.600979 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.600958    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.100944542 +0000 UTC m=+31.107355982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:38.601255 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.601434 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601404    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-config-volume\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.601642 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e170d8-e67b-4972-9e92-28b033c07336-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.601719 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601671    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb5d1d52-3223-45e9-a766-848d837c029b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.601841 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-service-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.601902 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601860    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.601951 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.601951 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601917    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.601951 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.602093 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.601965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aae52ad6-e69f-4db7-8924-3927e0b9af50-cert\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.602093 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.602000    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5998295-35aa-458b-9174-625f915c16cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:38.602206 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.602142    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.102102581 +0000 UTC m=+31.108514026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:38.602262 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.602213    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:38.602262 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.602234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-serving-cert\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.602262 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:38.602251    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:39.102239331 +0000 UTC m=+31.108650766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:38.602441 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.602412    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.602533 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.602457    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.603486 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.603464    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-config\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.603767 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.603741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.603886 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.603870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedb70cd-4f51-41ae-acfd-6c5c2044644d-serving-cert\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.604375 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.604356    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aae52ad6-e69f-4db7-8924-3927e0b9af50-cert\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.605660 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.605645    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fc2433-5459-47d9-a32f-511bf10bcf70-serving-cert\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.624475 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.624449    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 17:22:38.624475 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.624471    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xcdtc\""
Nov 02 17:22:38.624727 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.624713    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lh9zn\""
Nov 02 17:22:38.624727 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.624729    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b28rw\""
Nov 02 17:22:38.624861 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.624771    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-nxqk5\""
Nov 02 17:22:38.624861 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.624849    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vb5h7\""
Nov 02 17:22:38.625749 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.625732    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rn2gc"]
Nov 02 17:22:38.626467 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.626449    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d9d94667c-xzxd4"]
Nov 02 17:22:38.627065 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.627053    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"]
Nov 02 17:22:38.627702 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.627682    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"]
Nov 02 17:22:38.629541 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.629494    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nt8\""
Nov 02 17:22:38.629541 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.629501    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:22:38.629541 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.629514    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:22:38.629541 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.629542    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v5b9c\""
Nov 02 17:22:38.629712 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.629558    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l2dvj\""
Nov 02 17:22:38.631487 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.631465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.632758 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.632738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.650998 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.650965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kr6\" (UniqueName: \"kubernetes.io/projected/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-kube-api-access-f7kr6\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.651340 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.651325    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqnp\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.651851 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.651830    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdw4\" (UniqueName: \"kubernetes.io/projected/a1767839-4632-46c7-84aa-19269b3be129-kube-api-access-lzdw4\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:38.652258 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.652237    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmrr\" (UniqueName: \"kubernetes.io/projected/d269254c-0c37-4fad-b7e4-20b88ee2c2c1-kube-api-access-5hmrr\") pod \"kas-connection-checker-685895c7c9-tbpf9\" (UID: \"d269254c-0c37-4fad-b7e4-20b88ee2c2c1\") " pod="kube-system/kas-connection-checker-685895c7c9-tbpf9"
Nov 02 17:22:38.654810 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.654783    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjmg\" (UniqueName: \"kubernetes.io/projected/eedb70cd-4f51-41ae-acfd-6c5c2044644d-kube-api-access-qtjmg\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.657705 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.657682    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxz6p\" (UniqueName: \"kubernetes.io/projected/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-kube-api-access-rxz6p\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:38.658010 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.657994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:38.679359 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.679339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tbpf9"
Nov 02 17:22:38.686896 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.686836    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs858\" (UniqueName: \"kubernetes.io/projected/8a562913-9258-41b3-960e-c9dc07e2637c-kube-api-access-xs858\") pod \"kas-connection-checker-685895c7c9-gkwvc\" (UID: \"8a562913-9258-41b3-960e-c9dc07e2637c\") " pod="kube-system/kas-connection-checker-685895c7c9-gkwvc"
Nov 02 17:22:38.687065 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.687039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqw5\" (UniqueName: \"kubernetes.io/projected/c7260359-2132-4442-8388-5cd2455b5700-kube-api-access-mvqw5\") pod \"network-check-source-6f78d59f99-988dm\" (UID: \"c7260359-2132-4442-8388-5cd2455b5700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"
Nov 02 17:22:38.687266 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.687242    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.687398 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.687273    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflzw\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:38.687760 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.687744    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wxdc\" (UniqueName: \"kubernetes.io/projected/cd56f1da-de64-4541-9c26-e5ba4b221dfa-kube-api-access-9wxdc\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:38.687982 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.687959    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxgv\" (UniqueName: \"kubernetes.io/projected/f2fc2433-5459-47d9-a32f-511bf10bcf70-kube-api-access-5qxgv\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.688035 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.687960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/45fb7f7a-5ad6-4176-8405-ece930a49aca-kube-api-access-flcdx\") pod \"volume-data-source-validator-58c9f5f46d-6c5hm\" (UID: \"45fb7f7a-5ad6-4176-8405-ece930a49aca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"
Nov 02 17:22:38.688749 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.688723    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslwl\" (UniqueName: \"kubernetes.io/projected/44e170d8-e67b-4972-9e92-28b033c07336-kube-api-access-vslwl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.689516 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.689492    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsr5f\" (UniqueName: \"kubernetes.io/projected/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc-kube-api-access-bsr5f\") pod \"kas-connection-checker-685895c7c9-czpdz\" (UID: \"c8f53a9e-fe98-4056-8d6b-29bd6f1912cc\") " pod="kube-system/kas-connection-checker-685895c7c9-czpdz"
Nov 02 17:22:38.690086 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.690068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8mg\" (UniqueName: \"kubernetes.io/projected/aae52ad6-e69f-4db7-8924-3927e0b9af50-kube-api-access-rm8mg\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.690170 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.690151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskxm\" (UniqueName: \"kubernetes.io/projected/fb5d1d52-3223-45e9-a766-848d837c029b-kube-api-access-gskxm\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:38.691524 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.691505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gkwvc"
Nov 02 17:22:38.715265 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.715237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:38.731715 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.731687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:38.747549 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.746050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:38.757574 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.757518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"
Nov 02 17:22:38.770873 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.770843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:38.774371 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.774323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"
Nov 02 17:22:38.795915 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.795711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:38.984514 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:38.984488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-czpdz"
Nov 02 17:22:39.077891 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.077870    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gkwvc"]
Nov 02 17:22:39.080217 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.080193    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tbpf9"]
Nov 02 17:22:39.080922 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.080899    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a562913_9258_41b3_960e_c9dc07e2637c.slice/crio-422b11a23c76cc4203204d6ec081e37cfc5bbefe3fee0b5b9a966aa3ff33295b WatchSource:0}: Error finding container 422b11a23c76cc4203204d6ec081e37cfc5bbefe3fee0b5b9a966aa3ff33295b: Status 404 returned error can't find the container with id 422b11a23c76cc4203204d6ec081e37cfc5bbefe3fee0b5b9a966aa3ff33295b
Nov 02 17:22:39.083661 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.083641    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd269254c_0c37_4fad_b7e4_20b88ee2c2c1.slice/crio-5724f883979c6694a2ad377ab8ecfb6956e05490ba0758059d66ade9a63590a5 WatchSource:0}: Error finding container 5724f883979c6694a2ad377ab8ecfb6956e05490ba0758059d66ade9a63590a5: Status 404 returned error can't find the container with id 5724f883979c6694a2ad377ab8ecfb6956e05490ba0758059d66ade9a63590a5
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108603    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108674    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.108654053 +0000 UTC m=+32.115065489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108707    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108756    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.10874071 +0000 UTC m=+32.115152149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108796    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108809    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108842    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.108831377 +0000 UTC m=+32.115242819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108903    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:39.109241 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.108911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108939    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.108928263 +0000 UTC m=+32.115339684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108977    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.109010    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.108999159 +0000 UTC m=+32.115410604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.108602    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.109023    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.109029    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.109052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.109042251 +0000 UTC m=+32.115453713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:39.110151 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.109069    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.109059866 +0000 UTC m=+32.115471307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:39.110622 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:39.110603    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.11058653 +0000 UTC m=+32.116997956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:39.127957 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.127859    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"]
Nov 02 17:22:39.134394 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.134333    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedb70cd_4f51_41ae_acfd_6c5c2044644d.slice/crio-9f13b477b9da00199e7891b2c72d581648896986aab5a8961c87ab84c8ba421f WatchSource:0}: Error finding container 9f13b477b9da00199e7891b2c72d581648896986aab5a8961c87ab84c8ba421f: Status 404 returned error can't find the container with id 9f13b477b9da00199e7891b2c72d581648896986aab5a8961c87ab84c8ba421f
Nov 02 17:22:39.199470 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.199431    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"]
Nov 02 17:22:39.200856 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.200837    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"]
Nov 02 17:22:39.203900 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.203873    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-5hwls"]
Nov 02 17:22:39.208748 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.208720    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e170d8_e67b_4972_9e92_28b033c07336.slice/crio-92b1d840f560a9fbdb0b7e813ae4dfb7a2ab26de3719e3783f2517332082b781 WatchSource:0}: Error finding container 92b1d840f560a9fbdb0b7e813ae4dfb7a2ab26de3719e3783f2517332082b781: Status 404 returned error can't find the container with id 92b1d840f560a9fbdb0b7e813ae4dfb7a2ab26de3719e3783f2517332082b781
Nov 02 17:22:39.209515 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.209489    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fc2433_5459_47d9_a32f_511bf10bcf70.slice/crio-8287ae181d3c04c8903b980ec2970741ab565d526557147276b7e363af02f8c3 WatchSource:0}: Error finding container 8287ae181d3c04c8903b980ec2970741ab565d526557147276b7e363af02f8c3: Status 404 returned error can't find the container with id 8287ae181d3c04c8903b980ec2970741ab565d526557147276b7e363af02f8c3
Nov 02 17:22:39.209897 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.209875    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fb7f7a_5ad6_4176_8405_ece930a49aca.slice/crio-3402f0902ae94ac6a0fabd9a8d3ee2a7ac145dc55d280e87453c5682e28277ca WatchSource:0}: Error finding container 3402f0902ae94ac6a0fabd9a8d3ee2a7ac145dc55d280e87453c5682e28277ca: Status 404 returned error can't find the container with id 3402f0902ae94ac6a0fabd9a8d3ee2a7ac145dc55d280e87453c5682e28277ca
Nov 02 17:22:39.219970 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.219953    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"]
Nov 02 17:22:39.221881 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.221862    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5438c9_cd21_4ac3_93b6_44579f5b7b4e.slice/crio-1da67b4e721f3d8dae656bb034f26bc78b0676e8a89101b937d78b141c138c68 WatchSource:0}: Error finding container 1da67b4e721f3d8dae656bb034f26bc78b0676e8a89101b937d78b141c138c68: Status 404 returned error can't find the container with id 1da67b4e721f3d8dae656bb034f26bc78b0676e8a89101b937d78b141c138c68
Nov 02 17:22:39.241009 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.240989    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"]
Nov 02 17:22:39.243632 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.243608    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7260359_2132_4442_8388_5cd2455b5700.slice/crio-811104f5dd9df79256ed762587b28259775399b7def51b3a5438c4ebfd90e3c2 WatchSource:0}: Error finding container 811104f5dd9df79256ed762587b28259775399b7def51b3a5438c4ebfd90e3c2: Status 404 returned error can't find the container with id 811104f5dd9df79256ed762587b28259775399b7def51b3a5438c4ebfd90e3c2
Nov 02 17:22:39.272249 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.272231    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8v2j6"]
Nov 02 17:22:39.273976 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.273952    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae52ad6_e69f_4db7_8924_3927e0b9af50.slice/crio-f9c7beadad6fbce2382c7a9b41aec0cab972957ddac53527cecb985e4e668034 WatchSource:0}: Error finding container f9c7beadad6fbce2382c7a9b41aec0cab972957ddac53527cecb985e4e668034: Status 404 returned error can't find the container with id f9c7beadad6fbce2382c7a9b41aec0cab972957ddac53527cecb985e4e668034
Nov 02 17:22:39.356295 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.356218    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-czpdz"]
Nov 02 17:22:39.358553 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:39.358531    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f53a9e_fe98_4056_8d6b_29bd6f1912cc.slice/crio-4f686265663c1a8a47cc3ef60fe7583a1344992de02c63d1ba8200d73edf43fd WatchSource:0}: Error finding container 4f686265663c1a8a47cc3ef60fe7583a1344992de02c63d1ba8200d73edf43fd: Status 404 returned error can't find the container with id 4f686265663c1a8a47cc3ef60fe7583a1344992de02c63d1ba8200d73edf43fd
Nov 02 17:22:39.685340 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.685254    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8v2j6" event={"ID":"aae52ad6-e69f-4db7-8924-3927e0b9af50","Type":"ContainerStarted","Data":"f9c7beadad6fbce2382c7a9b41aec0cab972957ddac53527cecb985e4e668034"}
Nov 02 17:22:39.686421 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.686365    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm" event={"ID":"c7260359-2132-4442-8388-5cd2455b5700","Type":"ContainerStarted","Data":"811104f5dd9df79256ed762587b28259775399b7def51b3a5438c4ebfd90e3c2"}
Nov 02 17:22:39.687501 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.687477    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm" event={"ID":"45fb7f7a-5ad6-4176-8405-ece930a49aca","Type":"ContainerStarted","Data":"3402f0902ae94ac6a0fabd9a8d3ee2a7ac145dc55d280e87453c5682e28277ca"}
Nov 02 17:22:39.688497 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.688480    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl" event={"ID":"44e170d8-e67b-4972-9e92-28b033c07336","Type":"ContainerStarted","Data":"92b1d840f560a9fbdb0b7e813ae4dfb7a2ab26de3719e3783f2517332082b781"}
Nov 02 17:22:39.689491 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.689471    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" event={"ID":"eedb70cd-4f51-41ae-acfd-6c5c2044644d","Type":"ContainerStarted","Data":"9f13b477b9da00199e7891b2c72d581648896986aab5a8961c87ab84c8ba421f"}
Nov 02 17:22:39.690474 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.690453    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-5hwls" event={"ID":"f2fc2433-5459-47d9-a32f-511bf10bcf70","Type":"ContainerStarted","Data":"8287ae181d3c04c8903b980ec2970741ab565d526557147276b7e363af02f8c3"}
Nov 02 17:22:39.691720 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.691701    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gkwvc" event={"ID":"8a562913-9258-41b3-960e-c9dc07e2637c","Type":"ContainerStarted","Data":"45b239ba9bd6f39a546564de405c88c7c4a8a0c0889b4385cf7df617d2617197"}
Nov 02 17:22:39.691811 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.691731    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gkwvc" event={"ID":"8a562913-9258-41b3-960e-c9dc07e2637c","Type":"ContainerStarted","Data":"422b11a23c76cc4203204d6ec081e37cfc5bbefe3fee0b5b9a966aa3ff33295b"}
Nov 02 17:22:39.693271 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.693251    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tbpf9" event={"ID":"d269254c-0c37-4fad-b7e4-20b88ee2c2c1","Type":"ContainerStarted","Data":"1b50066e58c365ca155763951f72e934a7b9302f70cc5d3451c770f1e5190576"}
Nov 02 17:22:39.693349 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.693287    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tbpf9" event={"ID":"d269254c-0c37-4fad-b7e4-20b88ee2c2c1","Type":"ContainerStarted","Data":"5724f883979c6694a2ad377ab8ecfb6956e05490ba0758059d66ade9a63590a5"}
Nov 02 17:22:39.694663 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.694644    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-czpdz" event={"ID":"c8f53a9e-fe98-4056-8d6b-29bd6f1912cc","Type":"ContainerStarted","Data":"31df64ebd664b1913628c9f053fdc8762286bb3aba9826d8e4b00943aad5f67f"}
Nov 02 17:22:39.694663 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.694673    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-czpdz" event={"ID":"c8f53a9e-fe98-4056-8d6b-29bd6f1912cc","Type":"ContainerStarted","Data":"4f686265663c1a8a47cc3ef60fe7583a1344992de02c63d1ba8200d73edf43fd"}
Nov 02 17:22:39.695650 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:39.695632    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk" event={"ID":"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e","Type":"ContainerStarted","Data":"1da67b4e721f3d8dae656bb034f26bc78b0676e8a89101b937d78b141c138c68"}
Nov 02 17:22:40.119092 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:40.119092 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119229    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119294    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119295    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:40.119311 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119312    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119235    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119339    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119307    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.119287263 +0000 UTC m=+34.125698708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119378    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.11936653 +0000 UTC m=+34.125777952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119392    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.119383194 +0000 UTC m=+34.125794616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119408    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.11939984 +0000 UTC m=+34.125811266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.119471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119480    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119512    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.119502386 +0000 UTC m=+34.125913820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119526    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.119518455 +0000 UTC m=+34.125929883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:40.119536 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119539    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.119531944 +0000 UTC m=+34.125943369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:40.119870 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119563    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:40.119870 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119571    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:40.119870 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:40.119597    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.119591132 +0000 UTC m=+34.126002554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:40.717888 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.717834    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tbpf9" podStartSLOduration=208.717816033 podStartE2EDuration="3m28.717816033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:40.717210804 +0000 UTC m=+32.723622254" watchObservedRunningTime="2026-11-02 17:22:40.717816033 +0000 UTC m=+32.724227479"
Nov 02 17:22:40.718372 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:40.718327    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gkwvc" podStartSLOduration=208.718316225 podStartE2EDuration="3m28.718316225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:40.600742283 +0000 UTC m=+32.607153726" watchObservedRunningTime="2026-11-02 17:22:40.718316225 +0000 UTC m=+32.724727671"
Nov 02 17:22:41.433411 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.433371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:41.433593 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.433423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:41.433743 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:41.433717    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:41.433883 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:41.433787    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:23:13.433768263 +0000 UTC m=+65.440179699 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : secret "metrics-daemon-secret" not found
Nov 02 17:22:41.447552 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.447491    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:41.504524 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.504313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:41.534935 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.534896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:41.538799 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.538745    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:41.815850 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.815818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:41.994578 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.993561    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-czpdz" podStartSLOduration=209.993538989 podStartE2EDuration="3m29.993538989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:40.834588102 +0000 UTC m=+32.840999547" watchObservedRunningTime="2026-11-02 17:22:41.993538989 +0000 UTC m=+33.999950435"
Nov 02 17:22:41.994578 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:41.994467    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9m4h8"]
Nov 02 17:22:42.003534 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:42.003504    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d157fc_a912_44a3_9359_23691282dda2.slice/crio-a512ccda2add555fd9f2fc7981772530228dd8d45da853d17f170f901ef6a00d WatchSource:0}: Error finding container a512ccda2add555fd9f2fc7981772530228dd8d45da853d17f170f901ef6a00d: Status 404 returned error can't find the container with id a512ccda2add555fd9f2fc7981772530228dd8d45da853d17f170f901ef6a00d
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.141903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142045    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142106    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.142088872 +0000 UTC m=+38.148500298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142522    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142573    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.142557696 +0000 UTC m=+38.148969124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142643    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142654    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142686    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.142674988 +0000 UTC m=+38.149086418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:42.143136 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142743    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142773    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.14276337 +0000 UTC m=+38.149174806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142832    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142842    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.142862162 +0000 UTC m=+38.149273598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142929    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.142958    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.14294783 +0000 UTC m=+38.149359268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.143013    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.143035    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.143028858 +0000 UTC m=+38.149440292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:42.144060 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:42.143087    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.143081511 +0000 UTC m=+38.149492947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:42.259682 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.259297    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-przls"]
Nov 02 17:22:42.262140 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:42.262076    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b00385_648c_4505_a9a8_75070661432f.slice/crio-680bef45ba25f391b3064a17eb89b07a8a68b45e83f628dbfa139891b725ab7a WatchSource:0}: Error finding container 680bef45ba25f391b3064a17eb89b07a8a68b45e83f628dbfa139891b725ab7a: Status 404 returned error can't find the container with id 680bef45ba25f391b3064a17eb89b07a8a68b45e83f628dbfa139891b725ab7a
Nov 02 17:22:42.716481 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.716395    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9m4h8" event={"ID":"64d157fc-a912-44a3-9359-23691282dda2","Type":"ContainerStarted","Data":"a512ccda2add555fd9f2fc7981772530228dd8d45da853d17f170f901ef6a00d"}
Nov 02 17:22:42.722176 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.721377    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm" event={"ID":"45fb7f7a-5ad6-4176-8405-ece930a49aca","Type":"ContainerStarted","Data":"763ce743857f5a1472293e4c3e671c3936cb702104fc978da9019ca2ab9d2201"}
Nov 02 17:22:42.726134 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.726072    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-przls" event={"ID":"c0b00385-648c-4505-a9a8-75070661432f","Type":"ContainerStarted","Data":"680bef45ba25f391b3064a17eb89b07a8a68b45e83f628dbfa139891b725ab7a"}
Nov 02 17:22:42.840478 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:42.840415    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm" podStartSLOduration=177.474443533 podStartE2EDuration="2m59.840393839s" totalImagesPullingTime="2.365950306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:42.839921571 +0000 UTC m=+34.846333016" watchObservedRunningTime="2026-11-02 17:22:42.840393839 +0000 UTC m=+34.846805283"
Nov 02 17:22:44.734600 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:44.734578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/0.log"
Nov 02 17:22:44.735184 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:44.735162    2578 scope.go:122] "RemoveContainer" containerID="c3a84d63e68ff9b4961ac78a7956fe989005fcf3aae7be90f4ae44ff5b35ba70"
Nov 02 17:22:44.735734 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:44.735701    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" event={"ID":"eedb70cd-4f51-41ae-acfd-6c5c2044644d","Type":"ContainerDied","Data":"c3a84d63e68ff9b4961ac78a7956fe989005fcf3aae7be90f4ae44ff5b35ba70"}
Nov 02 17:22:44.735806 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:44.734613    2578 generic.go:356] "Generic (PLEG): container finished" podID="eedb70cd-4f51-41ae-acfd-6c5c2044644d" containerID="c3a84d63e68ff9b4961ac78a7956fe989005fcf3aae7be90f4ae44ff5b35ba70" exitCode=255
Nov 02 17:22:44.738900 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:44.738534    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-5hwls" event={"ID":"f2fc2433-5459-47d9-a32f-511bf10bcf70","Type":"ContainerStarted","Data":"e0ea498a5796cbd2452153523eefc80504d38e47c8d02c85227838c863027cb6"}
Nov 02 17:22:44.970285 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:44.970222    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-5hwls" podStartSLOduration=177.327417744 podStartE2EDuration="3m1.970203773s" totalImagesPullingTime="4.642786029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:44.969699537 +0000 UTC m=+36.976110975" watchObservedRunningTime="2026-11-02 17:22:44.970203773 +0000 UTC m=+36.976615230"
Nov 02 17:22:45.412200 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.409026    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8n69x"]
Nov 02 17:22:45.427956 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.427930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.442525 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.442432    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8n69x"]
Nov 02 17:22:45.481032 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.480827    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:22:45.481664 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.481627    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2zcw\""
Nov 02 17:22:45.482687 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.482662    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:22:45.508582 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.508542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33553579-055a-4dd9-986d-9e32c86d442f-data-volume\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.508747 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.508594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqw4\" (UniqueName: \"kubernetes.io/projected/33553579-055a-4dd9-986d-9e32c86d442f-kube-api-access-9kqw4\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.508747 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.508621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33553579-055a-4dd9-986d-9e32c86d442f-crio-socket\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.508747 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.508690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.508747 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.508735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.609390 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.609307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.609390 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.609393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.609643 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.609495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33553579-055a-4dd9-986d-9e32c86d442f-data-volume\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.609643 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.609521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqw4\" (UniqueName: \"kubernetes.io/projected/33553579-055a-4dd9-986d-9e32c86d442f-kube-api-access-9kqw4\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.609643 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.609550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33553579-055a-4dd9-986d-9e32c86d442f-crio-socket\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.609643 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.609641    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33553579-055a-4dd9-986d-9e32c86d442f-crio-socket\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.609867 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:45.609752    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:45.609867 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:45.609809    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.10978987 +0000 UTC m=+38.116201297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:45.610143 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:45.610096    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:45.610306 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:45.610187    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.110168762 +0000 UTC m=+38.116580184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:45.610396 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.610365    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33553579-055a-4dd9-986d-9e32c86d442f-data-volume\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.692909 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.692838    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqw4\" (UniqueName: \"kubernetes.io/projected/33553579-055a-4dd9-986d-9e32c86d442f-kube-api-access-9kqw4\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:45.749447 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.749409    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk" event={"ID":"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e","Type":"ContainerStarted","Data":"69905e2b52aab14abb6fc4f36dfc799525e40f0c632337e14304ac89d88fd5d5"}
Nov 02 17:22:45.751337 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.751310    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl" event={"ID":"44e170d8-e67b-4972-9e92-28b033c07336","Type":"ContainerStarted","Data":"cadb757f728e784e75d258be9ab60da79dd75b924a0c389bc28e3285edce2a87"}
Nov 02 17:22:45.753508 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.753484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/1.log"
Nov 02 17:22:45.754137 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.753990    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/0.log"
Nov 02 17:22:45.754137 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.754033    2578 generic.go:356] "Generic (PLEG): container finished" podID="eedb70cd-4f51-41ae-acfd-6c5c2044644d" containerID="6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818" exitCode=255
Nov 02 17:22:45.754283 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.754142    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" event={"ID":"eedb70cd-4f51-41ae-acfd-6c5c2044644d","Type":"ContainerDied","Data":"6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818"}
Nov 02 17:22:45.754501 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.754484    2578 scope.go:122] "RemoveContainer" containerID="c3a84d63e68ff9b4961ac78a7956fe989005fcf3aae7be90f4ae44ff5b35ba70"
Nov 02 17:22:45.754849 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.754598    2578 scope.go:122] "RemoveContainer" containerID="6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818"
Nov 02 17:22:45.754849 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:45.754825    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:22:45.868741 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:45.868687    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk" podStartSLOduration=176.955944839 podStartE2EDuration="3m2.868671995s" totalImagesPullingTime="5.912727156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:45.867022858 +0000 UTC m=+37.873434301" watchObservedRunningTime="2026-11-02 17:22:45.868671995 +0000 UTC m=+37.875083440"
Nov 02 17:22:46.106286 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.106232    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl" podStartSLOduration=177.643037592 podStartE2EDuration="3m3.106212441s" totalImagesPullingTime="5.463174849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:46.104534814 +0000 UTC m=+38.110946260" watchObservedRunningTime="2026-11-02 17:22:46.106212441 +0000 UTC m=+38.112623890"
Nov 02 17:22:46.112667 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.112641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:46.112842 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.112809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:46.112973 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.112900    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:46.112973 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.112962    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:46.112973 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.112968    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:47.112949594 +0000 UTC m=+39.119361022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:46.113182 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.113022    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:47.113008292 +0000 UTC m=+39.119419732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:46.213264 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:46.213424 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:46.213424 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:46.213424 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:46.213424 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:46.213424 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:46.213424 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213410    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213434    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213476    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213481    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213505    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213531    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213486    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.213466044 +0000 UTC m=+46.219877485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213494    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213571    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.213552767 +0000 UTC m=+46.219964197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213591    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.213581442 +0000 UTC m=+46.219992868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213621    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.213612616 +0000 UTC m=+46.220024043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213635    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.213628003 +0000 UTC m=+46.220039431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213658    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.213649656 +0000 UTC m=+46.220061084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:46.213666 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213677    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.213668831 +0000 UTC m=+46.220080253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:46.214172 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.213737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:46.214172 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213835    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:46.214172 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213847    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:46.214172 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.213885    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.21387262 +0000 UTC m=+46.220284066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:46.472219 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.472186    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"]
Nov 02 17:22:46.484980 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.484957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"
Nov 02 17:22:46.516837 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.516805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cwq8\" (UniqueName: \"kubernetes.io/projected/96e40d68-e813-41fe-8a8f-8c0e3e0cb312-kube-api-access-2cwq8\") pod \"migrator-66df5d496b-gqbmt\" (UID: \"96e40d68-e813-41fe-8a8f-8c0e3e0cb312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"
Nov 02 17:22:46.533351 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.533325    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"]
Nov 02 17:22:46.538479 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.538458    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xdlgk\""
Nov 02 17:22:46.548655 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.548634    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:46.548817 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.548801    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:22:46.617717 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.617538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwq8\" (UniqueName: \"kubernetes.io/projected/96e40d68-e813-41fe-8a8f-8c0e3e0cb312-kube-api-access-2cwq8\") pod \"migrator-66df5d496b-gqbmt\" (UID: \"96e40d68-e813-41fe-8a8f-8c0e3e0cb312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"
Nov 02 17:22:46.676449 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.676378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwq8\" (UniqueName: \"kubernetes.io/projected/96e40d68-e813-41fe-8a8f-8c0e3e0cb312-kube-api-access-2cwq8\") pod \"migrator-66df5d496b-gqbmt\" (UID: \"96e40d68-e813-41fe-8a8f-8c0e3e0cb312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"
Nov 02 17:22:46.761909 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.761870    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8v2j6" event={"ID":"aae52ad6-e69f-4db7-8924-3927e0b9af50","Type":"ContainerStarted","Data":"0c50bf91a00e879b961afd82aac3fa044f94178c520d841f5e44dc9d1b0b4583"}
Nov 02 17:22:46.764369 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.764305    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm" event={"ID":"c7260359-2132-4442-8388-5cd2455b5700","Type":"ContainerStarted","Data":"f0f9b8e6d95ee1b078a7048699ac8432d398a5329359afeaef35e80bdc6c741b"}
Nov 02 17:22:46.766178 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.766156    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/1.log"
Nov 02 17:22:46.766760 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.766584    2578 scope.go:122] "RemoveContainer" containerID="6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818"
Nov 02 17:22:46.766760 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:46.766724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:22:46.821465 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.821438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"
Nov 02 17:22:46.879988 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:46.879923    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8v2j6" podStartSLOduration=2.134200935 podStartE2EDuration="8.879900967s" totalImagesPullingTime="6.745700032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:46.878531808 +0000 UTC m=+38.884943253" watchObservedRunningTime="2026-11-02 17:22:46.879900967 +0000 UTC m=+38.886312418"
Nov 02 17:22:47.123848 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:47.123458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:47.123848 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:47.123633    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:47.123848 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:47.123652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:47.123848 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:47.123698    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:49.12368362 +0000 UTC m=+41.130095043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:47.123848 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:47.123723    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm" podStartSLOduration=176.993589556 podStartE2EDuration="3m4.123708933s" totalImagesPullingTime="7.130119377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:47.121605473 +0000 UTC m=+39.128016919" watchObservedRunningTime="2026-11-02 17:22:47.123708933 +0000 UTC m=+39.130120378"
Nov 02 17:22:47.123848 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:47.123772    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:47.123848 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:47.123819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:49.123806103 +0000 UTC m=+41.130217538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:47.217007 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:47.216927    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"]
Nov 02 17:22:47.220791 ip-10-0-129-72 kubenswrapper[2578]: W1102 17:22:47.220754    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e40d68_e813_41fe_8a8f_8c0e3e0cb312.slice/crio-d4542b150e8e5516f72fb1e148b725306e5a0bcd5eed441783f6235dc8d243b3 WatchSource:0}: Error finding container d4542b150e8e5516f72fb1e148b725306e5a0bcd5eed441783f6235dc8d243b3: Status 404 returned error can't find the container with id d4542b150e8e5516f72fb1e148b725306e5a0bcd5eed441783f6235dc8d243b3
Nov 02 17:22:47.771412 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:47.771367    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt" event={"ID":"96e40d68-e813-41fe-8a8f-8c0e3e0cb312","Type":"ContainerStarted","Data":"d4542b150e8e5516f72fb1e148b725306e5a0bcd5eed441783f6235dc8d243b3"}
Nov 02 17:22:48.715620 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:48.715587    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:48.715805 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:48.715662    2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:48.716096 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:48.716069    2578 scope.go:122] "RemoveContainer" containerID="6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818"
Nov 02 17:22:48.716359 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:48.716335    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:22:48.776366 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:48.776330    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-przls" event={"ID":"c0b00385-648c-4505-a9a8-75070661432f","Type":"ContainerStarted","Data":"927bdbc904e1dc1e2348f46797f15ceaadbc910dc0cc04d7e0807666f8772e02"}
Nov 02 17:22:48.776800 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:48.776511    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:48.897376 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:48.897318    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-przls" podStartSLOduration=34.896240028 podStartE2EDuration="40.897303663s" totalImagesPullingTime="6.001063635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:48.896137399 +0000 UTC m=+40.902548833" watchObservedRunningTime="2026-11-02 17:22:48.897303663 +0000 UTC m=+40.903715106"
Nov 02 17:22:49.145399 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:49.145299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:49.145556 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:49.145427    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:49.145556 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:49.145476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:49.145556 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:49.145488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:53.14547376 +0000 UTC m=+45.151885182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:49.145716 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:49.145572    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:49.145716 ip-10-0-129-72 kubenswrapper[2578]: E1102 17:22:49.145629    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:53.145614237 +0000 UTC m=+45.152025676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:49.541410 ip-10-0-129-72 kubenswrapper[2578]: I1102 17:22:49.541361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m7rhm_4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:49.660634 ip-10-0-129-72 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 17:22:49.674500 ip-10-0-129-72 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 17:22:49.674713 ip-10-0-129-72 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 17:22:49.674775 ip-10-0-129-72 systemd[1]: kubelet.service: Consumed 3.116s CPU time.
Nov 02 17:22:49.683654 ip-10-0-129-72 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:22:49.776689 ip-10-0-129-72 kubenswrapper[4956]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:49.776689 ip-10-0-129-72 kubenswrapper[4956]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:22:49.776689 ip-10-0-129-72 kubenswrapper[4956]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:49.776689 ip-10-0-129-72 kubenswrapper[4956]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:49.778843 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778755    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.778843 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778843    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.778843 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778847    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778850    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778853    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778856    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778860    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778864    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778868    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778871    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778874    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778877    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778880    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778883    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778889    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778892    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778895    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778897    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778900    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778903    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778905    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.778941 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778908    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778910    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778913    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778916    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778919    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778921    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778924    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778927    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778929    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778932    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778934    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778937    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778939    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778942    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778945    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778948    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778950    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778953    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778956    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.779490 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778959    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778962    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778964    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778967    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778969    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778972    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778974    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778977    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778979    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778982    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778984    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778987    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778990    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778992    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778995    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.778999    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779001    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779004    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779006    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779009    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.779948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779012    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779015    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779017    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779020    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779022    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779025    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779028    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779030    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779033    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779035    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779038    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779040    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779043    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779046    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779048    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779050    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779055    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779058    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779061    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779064    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.780444 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779066    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779069    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779071    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779074    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779077    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779079    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779082    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779085    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779088    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779091    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779093    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779096    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779099    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779102    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779104    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779108    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779110    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779113    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779128    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.780945 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779131    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779134    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779137    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779139    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779142    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779144    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779147    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779900    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779906    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779909    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779911    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779914    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779917    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779920    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779923    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779925    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779928    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779930    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779933    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779936    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.781472 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779938    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779941    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779943    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779946    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779948    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779951    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779954    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779956    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779959    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779962    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779964    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779967    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779970    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779973    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779975    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779978    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779981    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779983    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779986    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.782004 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779988    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779991    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779993    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779996    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.779998    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780001    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780003    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780006    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780008    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780011    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780013    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780016    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780018    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780020    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780023    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780025    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780028    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780030    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780033    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780036    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.782504 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780038    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780041    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780043    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780046    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780049    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780052    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780055    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780058    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780060    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780063    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780065    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780068    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780070    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780073    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780075    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780077    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780080    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780082    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780085    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.782994 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780087    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780090    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780094    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780099    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780101    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780104    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780107    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780110    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780112    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780130    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780132    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780136    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780138    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780141    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780143    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780146    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780149    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780152    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780155    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780158    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.783499 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780161    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780164    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780167    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780169    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780172    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780177    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780181    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780185    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780188    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780190    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780193    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780195    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780197    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780200    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780203    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780745    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780752    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780756    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780761    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.783983 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780764    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780767    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780769    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780771    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780774    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780777    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780781    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780785    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780788    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780794    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780797    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780799    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780802    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780806    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780809    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780811    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780814    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780817    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780819    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.784469 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780822    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780824    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780827    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780830    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780833    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780835    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780838    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780840    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780843    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780845    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780848    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780850    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780853    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780855    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780858    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780860    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780862    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780865    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780868    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780870    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.784926 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780873    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780875    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780879    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780881    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780884    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780886    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780889    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780892    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780894    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780897    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780899    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780902    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780905    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780907    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780910    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780913    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780917    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780919    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780922    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780925    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.785428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780927    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780930    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780932    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780935    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780937    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780940    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780942    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780945    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780947    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780950    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780952    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780955    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780957    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780960    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780963    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780966    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780968    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780971    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780973    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.786222 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780975    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780978    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780981    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780983    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780986    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780988    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780991    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780993    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.780996    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781000    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781003    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781005    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781008    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781010    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781012    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781015    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781017    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781022    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781024    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781027    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.787000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781029    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781032    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781034    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781037    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781145    4956 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781156    4956 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781165    4956 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781176    4956 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781182    4956 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781185    4956 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781189    4956 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781193    4956 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781197    4956 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781199    4956 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781204    4956 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781208    4956 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781211    4956 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781214    4956 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781217    4956 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781220    4956 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781222    4956 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781227    4956 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781232    4956 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:22:49.787568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781235    4956 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781238    4956 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781241    4956 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781245    4956 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781248    4956 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781252    4956 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781255    4956 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781258    4956 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781261    4956 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781264    4956 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781267    4956 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781270    4956 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781273    4956 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781277    4956 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781280    4956 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781283    4956 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781287    4956 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781290    4956 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781293    4956 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781297    4956 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781300    4956 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781304    4956 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781307    4956 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781310    4956 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:22:49.788106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781315    4956 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781318    4956 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781321    4956 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781324    4956 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781327    4956 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781330    4956 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781334    4956 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781337    4956 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781340    4956 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781343    4956 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781345    4956 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781349    4956 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781352    4956 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781356    4956 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781359    4956 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781362    4956 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781365    4956 flags.go:64] FLAG: --help="false"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781368    4956 flags.go:64] FLAG: --hostname-override="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781371    4956 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781374    4956 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781377    4956 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781380    4956 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781384    4956 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:22:49.788870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781386    4956 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781391    4956 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781393    4956 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781396    4956 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781399    4956 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781402    4956 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781405    4956 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781408    4956 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781411    4956 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781414    4956 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781421    4956 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781424    4956 flags.go:64] FLAG: --lock-file=""
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781427    4956 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781430    4956 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781433    4956 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781440    4956 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781443    4956 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781446    4956 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781448    4956 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781451    4956 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781455    4956 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781458    4956 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781460    4956 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781465    4956 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781468    4956 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:22:49.789471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781472    4956 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781475    4956 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781478    4956 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781481    4956 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781484    4956 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781487    4956 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781490    4956 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781493    4956 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781501    4956 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781504    4956 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781507    4956 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781510    4956 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781513    4956 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781516    4956 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781519    4956 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781522    4956 flags.go:64] FLAG: --port="10250"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781525    4956 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781528    4956 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0eb094e71a9f34389"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781531    4956 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781534    4956 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781537    4956 flags.go:64] FLAG: --register-node="true"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781540    4956 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781551    4956 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781554    4956 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781557    4956 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781560    4956 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:22:49.790068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781564    4956 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781567    4956 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781570    4956 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781573    4956 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781576    4956 flags.go:64] FLAG: --runonce="false"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781579    4956 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781582    4956 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781585    4956 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781588    4956 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781591    4956 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781594    4956 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781597    4956 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781600    4956 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781603    4956 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781609    4956 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781612    4956 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781615    4956 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781618    4956 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781621    4956 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781623    4956 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781629    4956 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781631    4956 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781634    4956 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781639    4956 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781642    4956 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:22:49.790785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781650    4956 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781654    4956 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781657    4956 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781661    4956 flags.go:64] FLAG: --v="2"
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781665    4956 flags.go:64] FLAG: --version="false"
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781669    4956 flags.go:64] FLAG: --vmodule=""
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781673    4956 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.781676    4956 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781820    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781823    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781826    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781829    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781831    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781834    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781836    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781839    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781841    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781844    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781847    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781851    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781855    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.791566 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781860    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781862    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781865    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781867    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781870    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781872    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781875    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781877    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781880    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781883    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781885    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.781888    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782320    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782334    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782339    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782342    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782345    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782347    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782350    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782352    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.792067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782355    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782357    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782360    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782362    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782365    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782367    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782370    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782372    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782375    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782377    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782380    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782382    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782386    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782388    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782391    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782393    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782395    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782398    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782401    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782403    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.792612 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782405    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782410    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782413    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782416    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782435    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782440    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782444    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782447    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782450    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782454    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782456    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782459    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782462    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782464    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782467    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782470    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782472    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782475    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782477    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.793288 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782480    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782483    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782486    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782488    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782490    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782493    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782495    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782498    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782500    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782502    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782505    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782508    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782510    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782513    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782516    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782518    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782521    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782523    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782526    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782529    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.793952 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782532    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782535    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782537    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782540    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782542    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782544    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782547    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782549    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782552    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782554    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782557    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782559    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782562    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782564    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.782594    4956 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782630    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.794684 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782635    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782638    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782641    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782644    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782646    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782649    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782652    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782656    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782660    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782663    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782665    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782668    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782670    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782673    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782675    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782678    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782681    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782684    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782687    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782689    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.795079 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782692    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782694    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782697    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782699    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782702    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782704    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782707    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782709    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782711    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782714    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782716    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782719    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782721    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782723    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782726    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782728    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782730    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782733    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782735    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.795667 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782738    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782740    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782742    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782745    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782747    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782752    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782754    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782757    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782759    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782762    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782765    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782769    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782773    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782776    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782779    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782783    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782785    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782788    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782790    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.796255 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782793    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782795    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782798    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782800    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782802    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782805    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782808    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782810    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782813    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782815    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782818    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782820    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782823    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782825    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782828    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782830    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782833    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782835    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782839    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.796773 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782841    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782844    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782846    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782849    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782851    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782855    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782857    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782862    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782865    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782867    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782870    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782872    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782875    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782877    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782880    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782882    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782884    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782887    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782889    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782891    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.797283 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782894    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782896    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782899    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782901    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782903    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782907    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782909    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.782912    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.788304    4956 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.788323    4956 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.788343    4956 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.788348    4956 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788414    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788421    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788424    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788427    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788431    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788434    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788437    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.797765 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788439    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788442    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788444    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788447    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788450    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788452    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788455    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788458    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788460    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788463    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788466    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788468    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788471    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788473    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788479    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788482    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788485    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788487    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788490    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788493    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.798321 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788496    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788498    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788501    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788504    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788507    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788509    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788512    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788514    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788517    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788519    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788522    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788524    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788527    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788530    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788532    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788535    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788537    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788540    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788542    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.798862 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788545    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788549    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788553    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788556    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788559    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788562    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788565    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788569    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788571    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788574    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788578    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788581    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788583    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788586    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788588    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788591    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788593    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788596    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788598    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.799625 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788601    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788604    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788606    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788608    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788611    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788613    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788616    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788618    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788620    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788623    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788626    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788629    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788631    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788634    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788637    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788640    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788642    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788645    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788647    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788650    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.800360 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788653    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788655    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788658    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788661    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788664    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788666    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788669    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788671    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788674    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788676    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788679    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788681    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788684    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788686    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788689    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788693    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788696    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788699    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788701    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788704    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.801065 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788707    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.788735    4956 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788873    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788878    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788882    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788884    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788887    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788891    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788894    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788897    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788899    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788902    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788905    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788907    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788910    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788913    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.801722 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788915    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788918    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788920    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788923    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788925    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788927    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788930    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788932    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788935    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788938    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788941    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788944    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788946    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788948    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788951    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788953    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788956    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788958    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788961    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788964    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.802292 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788966    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788969    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788971    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788973    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788976    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788979    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788981    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788984    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788986    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788988    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788991    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788993    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.788998    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789001    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789004    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789008    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789012    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789015    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789019    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.802802 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789021    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789024    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789026    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789029    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789031    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789034    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789036    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789039    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789041    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789044    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789046    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789051    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789054    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789056    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789059    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789062    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789064    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789066    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789069    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.803287 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789072    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789075    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789077    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789079    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789082    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789085    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789087    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789090    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789092    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789095    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789097    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789100    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789102    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789105    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789107    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789110    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789112    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789134    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789137    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.803740 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789141    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789143    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789146    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789148    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789151    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789154    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789157    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789160    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789162    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789165    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789167    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789170    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789172    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789176    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789178    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.789202    4956 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:49.804246 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789237    4956 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789242    4956 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789245    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789253    4956 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789256    4956 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789259    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789262    4956 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789264    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789267    4956 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789270    4956 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789272    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789275    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789277    4956 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789280    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789282    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789284    4956 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789287    4956 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789289    4956 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789292    4956 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789294    4956 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.804733 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789297    4956 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789301    4956 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789303    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789306    4956 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789308    4956 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789312    4956 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789314    4956 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789317    4956 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789319    4956 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789322    4956 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789325    4956 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789328    4956 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789331    4956 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789333    4956 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789336    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789339    4956 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789341    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789343    4956 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789346    4956 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789348    4956 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.805267 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789351    4956 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789353    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789356    4956 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789358    4956 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789361    4956 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789363    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789366    4956 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789370    4956 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789373    4956 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789376    4956 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789380    4956 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789383    4956 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789387    4956 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789390    4956 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789393    4956 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789395    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789398    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789401    4956 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789403    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.805745 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789406    4956 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789409    4956 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789411    4956 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789414    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789417    4956 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789419    4956 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789422    4956 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789425    4956 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789427    4956 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789429    4956 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789432    4956 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789435    4956 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789437    4956 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789440    4956 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789442    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789444    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789447    4956 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789449    4956 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789451    4956 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.806212 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789454    4956 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789456    4956 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789459    4956 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789461    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789464    4956 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789466    4956 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789468    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789473    4956 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789475    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789478    4956 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789481    4956 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789483    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789486    4956 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789489    4956 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789492    4956 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789495    4956 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789497    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789500    4956 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789503    4956 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.806715 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789506    4956 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789508    4956 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789511    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789513    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789515    4956 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789518    4956 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789520    4956 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789523    4956 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:49.789525    4956 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.789599    4956 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.790623    4956 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.790676    4956 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.790849    4956 server.go:1014] "Starting client certificate rotation"
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.790998    4956 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.791041    4956 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:17:08 +0000 UTC" deadline="2028-08-07 15:39:10.771939194 +0000 UTC"
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.791080    4956 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15454h16m20.980861296s"
Nov 02 17:22:49.807211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.791369    4956 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:49.807607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.792611    4956 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:49.807607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.793945    4956 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:22:49.807607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.795691    4956 log.go:25] "Validated CRI v1 image API"
Nov 02 17:22:49.807607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.796389    4956 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:22:49.807607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.798462    4956 fs.go:135] Filesystem UUIDs: map[1ffc1acf-636e-48b2-9474-2339c4325f09:/dev/nvme0n1p3 5040071e-edea-48b9-a752-040513b24909:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 17:22:49.807749 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.798482    4956 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14a17a4a79ac75507316c26c5d5444dd64134161c541975c36c57edc089db83a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14a17a4a79ac75507316c26c5d5444dd64134161c541975c36c57edc089db83a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1da67b4e721f3d8dae656bb034f26bc78b0676e8a89101b937d78b141c138c68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1da67b4e721f3d8dae656bb034f26bc78b0676e8a89101b937d78b141c138c68/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3402f0902ae94ac6a0fabd9a8d3ee2a7ac145dc55d280e87453c5682e28277ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3402f0902ae94ac6a0fabd9a8d3ee2a7ac145dc55d280e87453c5682e28277ca/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/422b11a23c76cc4203204d6ec081e37cfc5bbefe3fee0b5b9a966aa3ff33295b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/422b11a23c76cc4203204d6ec081e37cfc5bbefe3fee0b5b9a966aa3ff33295b/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46b28861eecd2b516f7ff22ba695762cf36c736aac395ec261c0467ef7c4191b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46b28861eecd2b516f7ff22ba695762cf36c736aac395ec261c0467ef7c4191b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4906b6dee53066d5e6f1336ddc39c5dd9fb95ce08587ff4af6fc36005115645f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4906b6dee53066d5e6f1336ddc39c5dd9fb95ce08587ff4af6fc36005115645f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a3bac5553253fb4f68decc088f6ade27aadfe0726774073c9bc74ea8008eed5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a3bac5553253fb4f68decc088f6ade27aadfe0726774073c9bc74ea8008eed5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ef688615f21b2e3464110833e350512b0db70dd90dbb041062bbadab3aa25ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ef688615f21b2e3464110833e350512b0db70dd90dbb041062bbadab3aa25ad/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f686265663c1a8a47cc3ef60fe7583a1344992de02c63d1ba8200d73edf43fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f686265663c1a8a47cc3ef60fe7583a1344992de02c63d1ba8200d73edf43fd/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5724f883979c6694a2ad377ab8ecfb6956e05490ba0758059d66ade9a63590a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5724f883979c6694a2ad377ab8ecfb6956e05490ba0758059d66ade9a63590a5/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/680bef45ba25f391b3064a17eb89b07a8a68b45e83f628dbfa139891b725ab7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/680bef45ba25f391b3064a17eb89b07a8a68b45e83f628dbfa139891b725ab7a/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7effc1e998a71350420a8aeef9b373b0ddfee241c408a1d4b2a890d2ae18ff0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7effc1e998a71350420a8aeef9b373b0ddfee241c408a1d4b2a890d2ae18ff0d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/811104f5dd9df79256ed762587b28259775399b7def51b3a5438c4ebfd90e3c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/811104f5dd9df79256ed762587b28259775399b7def51b3a5438c4ebfd90e3c2/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8287ae181d3c04c8903b980ec2970741ab565d526557147276b7e363af02f8c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8287ae181d3c04c8903b980ec2970741ab565d526557147276b7e363af02f8c3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92b1d840f560a9fbdb0b7e813ae4dfb7a2ab26de3719e3783f2517332082b781/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92b1d840f560a9fbdb0b7e813ae4dfb7a2ab26de3719e3783f2517332082b781/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e096c81b5a8a3e2a43d3433df16c8b6ce1bdd076f4f3ad43a3dddbb09c9c714/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e096c81b5a8a3e2a43d3433df16c8b6ce1bdd076f4f3ad43a3dddbb09c9c714/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f13b477b9da00199e7891b2c72d581648896986aab5a8961c87ab84c8ba421f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f13b477b9da00199e7891b2c72d581648896986aab5a8961c87ab84c8ba421f/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a512ccda2add555fd9f2fc7981772530228dd8d45da853d17f170f901ef6a00d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a512ccda2add555fd9f2fc7981772530228dd8d45da853d17f170f901ef6a00d/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4542b150e8e5516f72fb1e148b725306e5a0bcd5eed441783f6235dc8d243b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4542b150e8e5516f72fb1e148b725306e5a0bcd5eed441783f6235dc8d243b3/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9c7beadad6fbce2382c7a9b41aec0cab972957ddac53527cecb985e4e668034/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9c7beadad6fbce2382c7a9b41aec0cab972957ddac53527cecb985e4e668034/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb8780c28b5b1810eb66007fb0ae9ca771493aeb3ac75963abaf4f7acc702202/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb8780c28b5b1810eb66007fb0ae9ca771493aeb3ac75963abaf4f7acc702202/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e/volumes/kubernetes.io~projected/kube-api-access-f7kr6:{mountpoint:/var/lib/kubelet/pods/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e/volumes/kubernetes.io~projected/kube-api-access-f7kr6 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5/volumes/kubernetes.io~projected/kube-api-access-dp25t:{mountpoint:/var/lib/kubelet/pods/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5/volumes/kubernetes.io~projected/kube-api-access-dp25t major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33553579-055a-4dd9-986d-9e32c86d442f/volumes/kubernetes.io~projected/kube-api-access-9kqw4:{mountpoint:/var/lib/kubelet/pods/33553579-055a-4dd9-986d-9e32c86d442f/volumes/kubernetes.io~projected/kube-api-access-9kqw4 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34a35c68-067f-4fac-82f0-a7ef888b21fb/volumes/kubernetes.io~projected/kube-api-access-6ndvj:{mountpoint:/var/lib/kubelet/pods/34a35c68-067f-4fac-82f0-a7ef888b21fb/volumes/kubernetes.io~projected/kube-api-access-6ndvj major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e170d8-e67b-4972-9e92-28b033c07336/volumes/kubernetes.io~projected/kube-api-access-vslwl:{mountpoint:/var/lib/kubelet/pods/44e170d8-e67b-4972-9e92-28b033c07336/volumes/kubernetes.io~projected/kube-api-access-vslwl major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e170d8-e67b-4972-9e92-28b033c07336/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/44e170d8-e67b-4972-9e92-28b033c07336/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45fb7f7a-5ad6-4176-8405-ece930a49aca/volumes/kubernetes.io~projected/kube-api-access-flcdx:{mountpoint:/var/lib/kubelet/pods/45fb7f7a-5ad6-4176-8405-ece930a49aca/volumes/kubernetes.io~projected/kube-api-access-flcdx major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volumes/kubernetes.io~projected/kube-api-access-sbrbx:{mountpoint:/var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volumes/kubernetes.io~projected/kube-api-access-sbrbx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52674954-19d1-447d-ade5-d8655b4db351/volumes/kubernetes.io~projected/kube-api-access-5p75t:{mountpoint:/var/lib/kubelet/pods/52674954-19d1-447d-ade5-d8655b4db351/volumes/kubernetes.io~projected/kube-api-access-5p75t major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/538a6445-5d03-43a2-a41c-0c8e89a53553/volumes/kubernetes.io~projected/kube-api-access-hrvd6:{mountpoint:/var/lib/kubelet/pods/538a6445-5d03-43a2-a41c-0c8e89a53553/volumes/kubernetes.io~projected/kube-api-access-hrvd6 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d157fc-a912-44a3-9359-23691282dda2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/64d157fc-a912-44a3-9359-23691282dda2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78779265-8cc2-4286-8ad2-3aed34f341dd/volumes/kubernetes.io~projected/kube-api-access-bgvvm:{mountpoint:/var/lib/kubelet/pods/78779265-8cc2-4286-8ad2-3aed34f341dd/volumes/kubernetes.io~projected/kube-api-access-bgvvm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a562913-9258-41b3-960e-c9dc07e2637c/volumes/kubernetes.io~projected/kube-api-access-xs858:{mountpoint:/var/lib/kubelet/pods/8a562913-9258-41b3-960e-c9dc07e2637c/volumes/kubernetes.io~projected/kube-api-access-xs858 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96e40d68-e813-41fe-8a8f-8c0e3e0cb312/volumes/kubernetes.io~projected/kube-api-access-2cwq8:{mountpoint:/var/lib/kubelet/pods/96e40d68-e813-41fe-8a8f-8c0e3e0cb312/volumes/kubernetes.io~projected/kube-api-access-2cwq8 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1767839-4632-46c7-84aa-19269b3be129/volumes/kubernetes.io~projected/kube-api-access-lzdw4:{mountpoint:/var/lib/kubelet/pods/a1767839-4632-46c7-84aa-19269b3be129/volumes/kubernetes.io~projected/kube-api-access-lzdw4 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aae52ad6-e69f-4db7-8924-3927e0b9af50/volumes/kubernetes.io~projected/kube-api-access-rm8mg:{mountpoint:/var/lib/kubelet/pods/aae52ad6-e69f-4db7-8924-3927e0b9af50/volumes/kubernetes.io~projected/kube-api-access-rm8mg major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aae52ad6-e69f-4db7-8924-3927e0b9af50/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/aae52ad6-e69f-4db7-8924-3927e0b9af50/volumes/kubernetes.io~secret/cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0b00385-648c-4505-a9a8-75070661432f/volumes/kubernetes.io~projected/kube-api-access-jxvkj:{mountpoint:/var/lib/kubelet/pods/c0b00385-648c-4505-a9a8-75070661432f/volumes/kubernetes.io~projected/kube-api-access-jxvkj major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7260359-2132-4442-8388-5cd2455b5700/volumes/kubernetes.io~projected/kube-api-access-mvqw5:{mountpoint:/var/lib/kubelet/pods/c7260359-2132-4442-8388-5cd2455b5700/volumes/kubernetes.io~projected/kube-api-access-mvqw5 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc/volumes/kubernetes.io~projected/kube-api-access-bsr5f:{mountpoint:/var/lib/kubelet/pods/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc/volumes/kubernetes.io~projected/kube-api-access-bsr5f major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~projected/kube-api-access-9wxdc:{mountpoint:/var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~projected/kube-api-access-9wxdc major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d269254c-0c37-4fad-b7e4-20b88ee2c2c1/volumes/kubernetes.io~projected/kube-api-access-5hmrr:{mountpoint:/var/lib/kubelet/pods/d269254c-0c37-4fad-b7e4-20b88ee2c2c1/volumes/kubernetes.io~projected/kube-api-access-5hmrr major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8ec8714-849f-4245-b41e-cb84f8a4996f/volumes/kubernetes.io~projected/kube-api-access-mz2d4:{mountpoint:/var/lib/kubelet/pods/d8ec8714-849f-4245-b41e-cb84f8a4996f/volumes/kubernetes.io~projected/kube-api-access-mz2d4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~projected/kube-api-access-79dbg:{mountpoint:/var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~projected/kube-api-access-79dbg major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eedb70cd-4f51-41ae-acfd-6c5c2044644d/volumes/kubernetes.io~projected/kube-api-access-qtjmg:{mountpoint:/var/lib/kubelet/pods/eedb70cd-4f51-41ae-acfd-6c5c2044644d/volumes/kubernetes.io~projected/kube-api-access-qtjmg major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eedb70cd-4f51-41ae-acfd-6c5c2044644d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/eedb70cd-4f51-41ae-acfd-6c5c2044644d/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2fc2433-5459-47d9-a32f-511bf10bcf70/volumes/kubernetes.io~projected/kube-api-access-5qxgv:{mountpoint:/var/lib/kubelet/pods/f2fc2433-5459-47d9-a32f-511bf10bcf70/volumes/kubernetes.io~projected/kube-api-access-5qxgv major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2fc2433-5459-47d9-a32f-511bf10bcf70/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f2fc2433-5459-47d9-a32f-511bf10bcf70/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3/volumes/kubernetes.io~projected/kube-api-access-bwczd:{mountpoint:/var/lib/kubelet/pods/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3/volumes/kubernetes.io~projected/kube-api-access-bwczd major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b/volumes/kubernetes.io~projected/kube-api-access-rxz6p:{mountpoint:/var/lib/kubelet/pods/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b/volumes/kubernetes.io~projected/kube-api-access-rxz6p major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~projected/kube-api-access-cwqnp:{mountpoint:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~projected/kube-api-access-cwqnp major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb5d1d52-3223-45e9-a766-848d837c029b/volumes/kubernetes.io~projected/kube-api-access-gskxm:{mountpoint:/var/lib/kubelet/pods/fb5d1d52-3223-45e9-a766-848d837c029b/volumes/kubernetes.io~projected/kube-api-access-gskxm major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~projected/kube-api-access-mflzw:{mountpoint:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~projected/kube-api-access-mflzw major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2f6b605cdd5f650937800255c3bb64aeac33e6dab705c3c9c4d2c7b8d950e098/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/847dff785de09faa49eda2e2b8d30bbebe2fa44843cd251cf04e7fe5f54578e0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/41018436e1b68be5387ebd6e993a8f5794eae26d931c876352d22b6427d82dcb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3e90b40a01ecb030cb64bfa5a4c55fdf75204037f40e1c7d5ce0b9b254f6d353/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a45dad1bfcf4a6bcb07a573d0f78bf8dbf7701cc45d9f42213339d7b3b7261fc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2d07596a42f67bc276a01d921c22556c1a2f6afb1c20bee54938cd2c6e822627/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1366a5b8f30905ab2a5641e61c5e8fda584dba821caae23c49fb01f7d4a23928/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/22a2d30f5a6562744161785c778c5f33dbbd96bee8f43bf351ebf2c155699c61/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d56e6df8cb0140065075fa8019665967b099e84f0c46cd5e75052dd27d050756/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8f3a6631f82b6208eddb58a27e38b9e7b168112160236bde460ea2a2040ec986/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/25262e2cfdbdc2bf9faa929453854db4eb4b718f099f2fba4746dac1a8e97e85/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/f0ed53e170438c345a3a962fb4304d21ffe1fa81c6114f87832a4ab9f48ccfae/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0df871cd4afc0b72c15cab049baaf299bb3c5775660481fdd089edcac79edd6d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/43fe1820dd6bcf14dbcf5bdf5b4b332e2f0cef5f8abd9ffab8d02bfe15b8cc14/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1a3801cd5c044196b8704cc0ec5068537eaddff81f623a5759873fc1df72825e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a40b6c3b48e41168118470308cea53055da93d716154b2b68a93c9d4722b4027/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4190c1980ab3690b9225fd59c62d1dbc8f59fb5c6b40a750f8f3c563de2db214/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/eac98eccf2aa5fb82888b0c7f42b5fb6552477e923a1efe182ed52d1374d2ca0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/97e4ea7451de759e2120cf738750428c495b025bf1536b5e11afb7f6e8529556/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/deef9e49466db12dd3fb560495a4f001e31766482e46ecd7213e5e26806d3c60/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6be117ff03f6c04def115ddc073f3b901733fe1272de76ad4b93011a0bb20366/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b5477d1a4777d8ad5546a84cef0194b88cd91211dbffef7cf760091dc3a77d05/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/847324a9c369c393bde5dd6d8e58161b21c01919160e149071ac3a05ab79d110/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/5fd14d829dab1aa11b224e94cb90913cd45592cfad65baa54cda5ab57e66b800/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/593ae2cddbd2b39a52f301c96d92a5b6b9ca69f555e8b68847385689140d4ecf/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/69ffa0cc3b065e45c9ccec67c677b2634c9754829361a6d4adab1998133e80af/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/91eb3a925b21c676acf324dbd265a3324d23d7ff7221347118fd2c3e6823631c/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/90ce1539406c950665a8ca98afff57b5fe6569de5e3489286ab777b729bbe8e8/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/c5d8a72f86be9765b232719717d36ea7cb8bbe3df4543905e6d9fc452f6f937c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/da120f9c95f35b8ab342497d454e81e379f6a16ec1d1cfe847f38c474ab0c751/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/c1a146c2e526ff1ce8b29249fbc9e798a549a405df188d804cdfec111c086049/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/150b0367526755fe0f5d44f7d13204dcdcf2405cb8b8e062bb3467825eaba2dd/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/24b7fa30121b4a66f10716194f9f524a1dcb2c0139b3dd490cbd29f2c74c4e05/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/27e0628f96d1adb5b613d61ac5f8fd1499da4854416045371a730bc928694353/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/f52e7654a5102ccab507e83b7e04ba45c72ff28410ee6c0612c75afd3367b43b/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/6f7898691037745389c1c29f59bece99bfb0b8061dd9b75844216e4b37eb0b7d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/bc3a7aa903fbebf30280959467c6bce638f9271420659602fc10c2a57adca2db/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/5b8b40ea2734903fb62061419238316ba2ae9b6e6bdbc7c0cdb2ab4406525d4d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6e61dd5bcc5ae697f0ceb2c234f88f26a9a8a5e6b8fe83bf14c563d43d4f118d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/e1cd6f555473b0e616e00761c09d5d2a0194f2e23a637cd37a9a158aa49e8efb/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/73a47d8f6a607e2fc16b1df3c6935249819a36a0e422fcfac5891f664acf53c7/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/443b289f5756913acb6e831ec5aaef890c147d07d32b766bf0ef076c2d3772f0/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/c893abe3f07107227b1704d56533efd81865fd1e19a12486e094391a5130114f/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/31909f2973861917adbbc106681ab6b5909586512d35d07a8eb9e4a02fa555e6/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/24e39718dbbe552fa664e9bd640cf3b965ff024cd36f64fbad75ba91d475976b/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/a2c757e0687feb41b0b935341326e3e89e1bc9253144bcca5e49791f71c6bf29/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/cb1cc3c2e86a0a61584d1bc77bf6495ffaa360c3993cbf82a3272deb77d1c582/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/d159f30bb93e8e7bddd63a510c7e54bda8d6ce2b2ec86732c69276638aac9f3e/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/68a33a9d5bd4a9b39ccb17b89937f88aa30f350c6fd0dcd4747fbe28cd849815/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6e035f7d002f1d35d7cffb504b03f4dba5f706e0b0c884377aa85ef546dd5c81/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/20f442a319a279e5151c3f7871e0d8351f772724f6a9d2b5174832a7bd6f5fb9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5815315a7ca553cf6a26d830e39f2f5c898a3b1c783ff4723114c732a1b8d011/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/34201a6d90e032991384df6161b12513ebe8b639d696de0c223677098aef7803/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d60d095aa8e5e10b15d83938d0111a0c4b0cf9dc3ec0cc21ab3f13213c1eee01/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/58dadbf5516dfc06ce983a28e30ed9c49e8d85904d1fb129f6c88400cc19390e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/38d5311d691f83d05fefcf985da4a4e873eda6b0f3884b633a1e1f2692fecc0a/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 17:22:49.808022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.807299    4956 manager.go:217] Machine: {Timestamp:2026-11-02 17:22:49.806796973 +0000 UTC m=+0.063827584 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099396 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25ed9c7d79be194a99120b3e6c72e1 SystemUUID:ec25ed9c-7d79-be19-4a99-120b3e6c72e1 BootID:25b6b15b-6e60-47be-ad3d-de28f10ff2f2 Filesystems:[{Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9c7beadad6fbce2382c7a9b41aec0cab972957ddac53527cecb985e4e668034/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f2fc2433-5459-47d9-a32f-511bf10bcf70/volumes/kubernetes.io~projected/kube-api-access-5qxgv DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a3bac5553253fb4f68decc088f6ade27aadfe0726774073c9bc74ea8008eed5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f686265663c1a8a47cc3ef60fe7583a1344992de02c63d1ba8200d73edf43fd/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~projected/kube-api-access-9wxdc DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33553579-055a-4dd9-986d-9e32c86d442f/volumes/kubernetes.io~projected/kube-api-access-9kqw4 DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a512ccda2add555fd9f2fc7981772530228dd8d45da853d17f170f901ef6a00d/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5/volumes/kubernetes.io~projected/kube-api-access-dp25t DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3/volumes/kubernetes.io~projected/kube-api-access-bwczd DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~projected/kube-api-access-79dbg DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e/volumes/kubernetes.io~projected/kube-api-access-f7kr6 DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4542b150e8e5516f72fb1e148b725306e5a0bcd5eed441783f6235dc8d243b3/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/34a35c68-067f-4fac-82f0-a7ef888b21fb/volumes/kubernetes.io~projected/kube-api-access-6ndvj DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d8ec8714-849f-4245-b41e-cb84f8a4996f/volumes/kubernetes.io~projected/kube-api-access-mz2d4 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d269254c-0c37-4fad-b7e4-20b88ee2c2c1/volumes/kubernetes.io~projected/kube-api-access-5hmrr DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3402f0902ae94ac6a0fabd9a8d3ee2a7ac145dc55d280e87453c5682e28277ca/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/538a6445-5d03-43a2-a41c-0c8e89a53553/volumes/kubernetes.io~projected/kube-api-access-hrvd6 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7effc1e998a71350420a8aeef9b373b0ddfee241c408a1d4b2a890d2ae18ff0d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44e170d8-e67b-4972-9e92-28b033c07336/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8287ae181d3c04c8903b980ec2970741ab565d526557147276b7e363af02f8c3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1767839-4632-46c7-84aa-19269b3be129/volumes/kubernetes.io~projected/kube-api-access-lzdw4 DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44e170d8-e67b-4972-9e92-28b033c07336/volumes/kubernetes.io~projected/kube-api-access-vslwl DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb5d1d52-3223-45e9-a766-848d837c029b/volumes/kubernetes.io~projected/kube-api-access-gskxm DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14a17a4a79ac75507316c26c5d5444dd64134161c541975c36c57edc089db83a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volumes/kubernetes.io~projected/kube-api-access-sbrbx DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b/volumes/kubernetes.io~projected/kube-api-access-rxz6p DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aae52ad6-e69f-4db7-8924-3927e0b9af50/volumes/kubernetes.io~projected/kube-api-access-rm8mg DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e096c81b5a8a3e2a43d3433df16c8b6ce1bdd076f4f3ad43a3dddbb09c9c714/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/811104f5dd9df79256ed762587b28259775399b7def51b3a5438c4ebfd90e3c2/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd56f1da-de64-4541-9c26-e5ba4b221dfa/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eedb70cd-4f51-41ae-acfd-6c5c2044644d/volumes/kubernetes.io~projected/kube-api-access-qtjmg DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~projected/kube-api-access-cwqnp DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78779265-8cc2-4286-8ad2-3aed34f341dd/volumes/kubernetes.io~projected/kube-api-access-bgvvm DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a562913-9258-41b3-960e-c9dc07e2637c/volumes/kubernetes.io~projected/kube-api-access-xs858 DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7260359-2132-4442-8388-5cd2455b5700/volumes/kubernetes.io~projected/kube-api-access-mvqw5 DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45fb7f7a-5ad6-4176-8405-ece930a49aca/volumes/kubernetes.io~projected/kube-api-access-flcdx DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc/volumes/kubernetes.io~projected/kube-api-access-bsr5f DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0b00385-648c-4505-a9a8-75070661432f/volumes/kubernetes.io~projected/kube-api-access-jxvkj DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46b28861eecd2b516f7ff22ba695762cf36c736aac395ec261c0467ef7c4191b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5724f883979c6694a2ad377ab8ecfb6956e05490ba0758059d66ade9a63590a5/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92b1d840f560a9fbdb0b7e813ae4dfb7a2ab26de3719e3783f2517332082b781/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb8780c28b5b1810eb66007fb0ae9ca771493aeb3ac75963abaf4f7acc702202/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aae52ad6-e69f-4db7-8924-3927e0b9af50/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/481d442d-4aa0-40bb-b723-ed569f0386ca/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/422b11a23c76cc4203204d6ec081e37cfc5bbefe3fee0b5b9a966aa3ff33295b/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f13b477b9da00199e7891b2c72d581648896986aab5a8961c87ab84c8ba421f/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64d157fc-a912-44a3-9359-23691282dda2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4906b6dee53066d5e6f1336ddc39c5dd9fb95ce08587ff4af6fc36005115645f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1da67b4e721f3d8dae656bb034f26bc78b0676e8a89101b937d78b141c138c68/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de90b397-c8f3-4b8e-b89a-822b7bc465bd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ef688615f21b2e3464110833e350512b0db70dd90dbb041062bbadab3aa25ad/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52674954-19d1-447d-ade5-d8655b4db351/volumes/kubernetes.io~projected/kube-api-access-5p75t DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eedb70cd-4f51-41ae-acfd-6c5c2044644d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/680bef45ba25f391b3064a17eb89b07a8a68b45e83f628dbfa139891b725ab7a/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/96e40d68-e813-41fe-8a8f-8c0e3e0cb312/volumes/kubernetes.io~projected/kube-api-access-2cwq8 DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~projected/kube-api-access-mflzw DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f2fc2433-5459-47d9-a32f-511bf10bcf70/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1da67b4e721f3d8 MacAddress:1a:c3:f0:b2:d3:5e Speed:10000 Mtu:8901} {Name:3402f0902ae94ac MacAddress:8e:0e:b6:6a:d9:5b Speed:10000 Mtu:8901} {Name:422b11a23c76cc4 MacAddress:16:38:90:16:3e:7c Speed:10000 Mtu:8901} {Name:4f686265663c1a8 MacAddress:9a:9d:d3:a8:0b:32 Speed:10000 Mtu:8901} {Name:5724f883979c669 MacAddress:6a:9a:91:ce:f8:c8 Speed:10000 Mtu:8901} {Name:680bef45ba25f39 MacAddress:7e:e1:c9:49:02:8f Speed:10000 Mtu:8901} {Name:811104f5dd9df79 MacAddress:d2:93:b7:35:05:70 Speed:10000 Mtu:8901} {Name:8287ae181d3c04c MacAddress:d6:35:55:d6:98:39 Speed:10000 Mtu:8901} {Name:92b1d840f560a9f MacAddress:c6:0a:5e:5b:f4:e4 Speed:10000 Mtu:8901} {Name:9f13b477b9da001 MacAddress:d6:f1:9e:dd:37:dc Speed:10000 Mtu:8901} {Name:a512ccda2add555 MacAddress:3a:e6:89:a9:d8:cd Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3c:95:67:ee:31 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:d1:61:2e:1a:10 Speed:0 Mtu:8901} {Name:d4542b150e8e551 MacAddress:be:c3:21:36:60:16 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3c:95:67:ee:31 Speed:0 Mtu:9001} {Name:f9c7beadad6fbce MacAddress:46:2a:e0:f5:d4:7a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:c8:4a:2f:51:48 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:7c:3f:e4:d5:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:22:49.808022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.807698    4956 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:22:49.808022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.807754    4956 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:22:49.808022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808008    4956 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:22:49.808306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808029    4956 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-72.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:22:49.808306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808188    4956 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:22:49.808306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808197    4956 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:22:49.808306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808208    4956 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:49.808306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808228    4956 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:49.808306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808247    4956 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:22:49.808485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808414    4956 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:22:49.808526 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808516    4956 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:22:49.808558 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808554    4956 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:22:49.808590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808568    4956 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:22:49.808590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808577    4956 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:22:49.808590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808587    4956 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:22:49.808715 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.808595    4956 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:22:49.809530 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.809513    4956 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:22:49.809530 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.809531    4956 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:49.810223 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.810203    4956 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:22:49.812827 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.812811    4956 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:22:49.812827 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.812826    4956 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:49.812960 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.812951    4956 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 17:22:49.813174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813162    4956 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:22:49.813207 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813194    4956 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:22:49.813295 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813287    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:22:49.813325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813300    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:22:49.813325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813307    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:22:49.813325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813313    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:22:49.813325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813319    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:22:49.813325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813325    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813331    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813337    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813344    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813350    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813358    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813368    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813392    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:22:49.813482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.813398    4956 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:22:49.817009 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.816992    4956 server.go:1263] "Started kubelet"
Nov 02 17:22:49.817101 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.817069    4956 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:22:49.817253 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.817166    4956 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:22:49.817394 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.817283    4956 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:22:49.817634 ip-10-0-129-72 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:22:49.819560 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.819512    4956 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:22:49.820858 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.820842    4956 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:22:49.822542 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.822424    4956 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:22:49.822542 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.822445    4956 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:22:49.822668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.822627    4956 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:09 +0000 UTC" deadline="2028-04-28 08:14:08.196726551 +0000 UTC"
Nov 02 17:22:49.822723 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.822674    4956 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13022h51m18.374057927s"
Nov 02 17:22:49.822723 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.822698    4956 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:22:49.822723 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.822711    4956 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:22:49.823036 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.822732    4956 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-72.us-west-1.compute.internal\" not found"
Nov 02 17:22:49.823036 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.822842    4956 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:22:49.825054 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.825029    4956 factory.go:55] Registering systemd factory
Nov 02 17:22:49.825054 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.825055    4956 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.825834    4956 factory.go:153] Registering CRI-O factory
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.825856    4956 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.825933    4956 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.825953    4956 factory.go:103] Registering Raw factory
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.825971    4956 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.826371    4956 manager.go:319] Starting recovery of all containers
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827277    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8ec8714-849f-4245-b41e-cb84f8a4996f" volumeName="kubernetes.io/projected/d8ec8714-849f-4245-b41e-cb84f8a4996f-kube-api-access-mz2d4" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827313    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" volumeName="kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827330    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8ec8714-849f-4245-b41e-cb84f8a4996f" volumeName="kubernetes.io/configmap/d8ec8714-849f-4245-b41e-cb84f8a4996f-serviceca" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827342    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de90b397-c8f3-4b8e-b89a-822b7bc465bd" volumeName="kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-tmp" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827355    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2fc2433-5459-47d9-a32f-511bf10bcf70" volumeName="kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-tmp" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827367    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45fb7f7a-5ad6-4176-8405-ece930a49aca" volumeName="kubernetes.io/projected/45fb7f7a-5ad6-4176-8405-ece930a49aca-kube-api-access-flcdx" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827380    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78779265-8cc2-4286-8ad2-3aed34f341dd" volumeName="kubernetes.io/projected/78779265-8cc2-4286-8ad2-3aed34f341dd-kube-api-access-bgvvm" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827391    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" volumeName="kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827405    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" volumeName="kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827417    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" volumeName="kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827432    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e170d8-e67b-4972-9e92-28b033c07336" volumeName="kubernetes.io/configmap/44e170d8-e67b-4972-9e92-28b033c07336-config" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827445    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481d442d-4aa0-40bb-b723-ed569f0386ca" volumeName="kubernetes.io/projected/481d442d-4aa0-40bb-b723-ed569f0386ca-kube-api-access-sbrbx" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827459    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538a6445-5d03-43a2-a41c-0c8e89a53553" volumeName="kubernetes.io/projected/538a6445-5d03-43a2-a41c-0c8e89a53553-kube-api-access-hrvd6" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827471    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1767839-4632-46c7-84aa-19269b3be129" volumeName="kubernetes.io/configmap/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-config" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827488    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5998295-35aa-458b-9174-625f915c16cf" volumeName="kubernetes.io/configmap/a5998295-35aa-458b-9174-625f915c16cf-nginx-conf" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827500    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de90b397-c8f3-4b8e-b89a-822b7bc465bd" volumeName="kubernetes.io/projected/de90b397-c8f3-4b8e-b89a-822b7bc465bd-kube-api-access-79dbg" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827513    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eedb70cd-4f51-41ae-acfd-6c5c2044644d" volumeName="kubernetes.io/secret/eedb70cd-4f51-41ae-acfd-6c5c2044644d-serving-cert" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827526    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5438c9-cd21-4ac3-93b6-44579f5b7b4e" volumeName="kubernetes.io/secret/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-serving-cert" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827539    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5" volumeName="kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827551    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5" volumeName="kubernetes.io/projected/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-kube-api-access-dp25t" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827563    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aae52ad6-e69f-4db7-8924-3927e0b9af50" volumeName="kubernetes.io/projected/aae52ad6-e69f-4db7-8924-3927e0b9af50-kube-api-access-rm8mg" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827575    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d269254c-0c37-4fad-b7e4-20b88ee2c2c1" volumeName="kubernetes.io/projected/d269254c-0c37-4fad-b7e4-20b88ee2c2c1-kube-api-access-5hmrr" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827588    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eedb70cd-4f51-41ae-acfd-6c5c2044644d" volumeName="kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-trusted-ca" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827600    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2fc2433-5459-47d9-a32f-511bf10bcf70" volumeName="kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-snapshots" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827612    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" volumeName="kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827624    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e40d68-e813-41fe-8a8f-8c0e3e0cb312" volumeName="kubernetes.io/projected/96e40d68-e813-41fe-8a8f-8c0e3e0cb312-kube-api-access-2cwq8" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827659    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eedb70cd-4f51-41ae-acfd-6c5c2044644d" volumeName="kubernetes.io/projected/eedb70cd-4f51-41ae-acfd-6c5c2044644d-kube-api-access-qtjmg" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827671    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33553579-055a-4dd9-986d-9e32c86d442f" volumeName="kubernetes.io/projected/33553579-055a-4dd9-986d-9e32c86d442f-kube-api-access-9kqw4" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827700    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd56f1da-de64-4541-9c26-e5ba4b221dfa" volumeName="kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-default-certificate" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827712    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" volumeName="kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827725    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4627fcb-ebee-4c41-9d5b-ee5c0f61624b" volumeName="kubernetes.io/configmap/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-config-volume" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827737    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5438c9-cd21-4ac3-93b6-44579f5b7b4e" volumeName="kubernetes.io/configmap/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-config" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827748    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e170d8-e67b-4972-9e92-28b033c07336" volumeName="kubernetes.io/secret/44e170d8-e67b-4972-9e92-28b033c07336-serving-cert" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827760    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481d442d-4aa0-40bb-b723-ed569f0386ca" volumeName="kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827773    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52674954-19d1-447d-ade5-d8655b4db351" volumeName="kubernetes.io/projected/52674954-19d1-447d-ade5-d8655b4db351-kube-api-access-5p75t" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827785    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a562913-9258-41b3-960e-c9dc07e2637c" volumeName="kubernetes.io/projected/8a562913-9258-41b3-960e-c9dc07e2637c-kube-api-access-xs858" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827796    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1767839-4632-46c7-84aa-19269b3be129" volumeName="kubernetes.io/projected/a1767839-4632-46c7-84aa-19269b3be129-kube-api-access-lzdw4" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827807    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" volumeName="kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827819    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" volumeName="kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827830    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52674954-19d1-447d-ade5-d8655b4db351" volumeName="kubernetes.io/empty-dir/52674954-19d1-447d-ade5-d8655b4db351-tmp-dir" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827844    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f53a9e-fe98-4056-8d6b-29bd6f1912cc" volumeName="kubernetes.io/projected/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc-kube-api-access-bsr5f" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827856    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2fc2433-5459-47d9-a32f-511bf10bcf70" volumeName="kubernetes.io/projected/f2fc2433-5459-47d9-a32f-511bf10bcf70-kube-api-access-5qxgv" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827867    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33553579-055a-4dd9-986d-9e32c86d442f" volumeName="kubernetes.io/empty-dir/33553579-055a-4dd9-986d-9e32c86d442f-data-volume" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827879    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34a35c68-067f-4fac-82f0-a7ef888b21fb" volumeName="kubernetes.io/projected/34a35c68-067f-4fac-82f0-a7ef888b21fb-kube-api-access-6ndvj" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827890    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0b00385-648c-4505-a9a8-75070661432f" volumeName="kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827902    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7260359-2132-4442-8388-5cd2455b5700" volumeName="kubernetes.io/projected/c7260359-2132-4442-8388-5cd2455b5700-kube-api-access-mvqw5" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827916    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de90b397-c8f3-4b8e-b89a-822b7bc465bd" volumeName="kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-tuned" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827930    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eedb70cd-4f51-41ae-acfd-6c5c2044644d" volumeName="kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-config" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827941    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4627fcb-ebee-4c41-9d5b-ee5c0f61624b" volumeName="kubernetes.io/empty-dir/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-tmp-dir" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827985    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5" volumeName="kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-daemon-config" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.827999    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5" volumeName="kubernetes.io/secret/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-agent-certs" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828012    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd56f1da-de64-4541-9c26-e5ba4b221dfa" volumeName="kubernetes.io/projected/cd56f1da-de64-4541-9c26-e5ba4b221dfa-kube-api-access-9wxdc" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828025    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2fc2433-5459-47d9-a32f-511bf10bcf70" volumeName="kubernetes.io/secret/f2fc2433-5459-47d9-a32f-511bf10bcf70-serving-cert" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828036    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" volumeName="kubernetes.io/projected/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-kube-api-access-bwczd" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828047    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" volumeName="kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828059    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2fc2433-5459-47d9-a32f-511bf10bcf70" volumeName="kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-service-ca-bundle" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828082    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4627fcb-ebee-4c41-9d5b-ee5c0f61624b" volumeName="kubernetes.io/projected/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-kube-api-access-rxz6p" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828095    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" volumeName="kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828109    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e170d8-e67b-4972-9e92-28b033c07336" volumeName="kubernetes.io/projected/44e170d8-e67b-4972-9e92-28b033c07336-kube-api-access-vslwl" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828138    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481d442d-4aa0-40bb-b723-ed569f0386ca" volumeName="kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-config" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828150    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aae52ad6-e69f-4db7-8924-3927e0b9af50" volumeName="kubernetes.io/secret/aae52ad6-e69f-4db7-8924-3927e0b9af50-cert" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828162    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" volumeName="kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828173    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34a35c68-067f-4fac-82f0-a7ef888b21fb" volumeName="kubernetes.io/configmap/34a35c68-067f-4fac-82f0-a7ef888b21fb-iptables-alerter-script" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828183    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481d442d-4aa0-40bb-b723-ed569f0386ca" volumeName="kubernetes.io/secret/481d442d-4aa0-40bb-b723-ed569f0386ca-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828196    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2fc2433-5459-47d9-a32f-511bf10bcf70" volumeName="kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828208    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" volumeName="kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828216    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5d1d52-3223-45e9-a766-848d837c029b" volumeName="kubernetes.io/projected/fb5d1d52-3223-45e9-a766-848d837c029b-kube-api-access-gskxm" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828223    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" volumeName="kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828230    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" volumeName="kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828237    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5438c9-cd21-4ac3-93b6-44579f5b7b4e" volumeName="kubernetes.io/projected/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-kube-api-access-f7kr6" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828246    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="481d442d-4aa0-40bb-b723-ed569f0386ca" volumeName="kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-env-overrides" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828254    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d157fc-a912-44a3-9359-23691282dda2" volumeName="kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828266    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" volumeName="kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828278    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" volumeName="kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828289    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5d1d52-3223-45e9-a766-848d837c029b" volumeName="kubernetes.io/configmap/fb5d1d52-3223-45e9-a766-848d837c029b-telemetry-config" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828301    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" volumeName="kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828315    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5" volumeName="kubernetes.io/configmap/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-konnectivity-ca" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828329    4956 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd56f1da-de64-4541-9c26-e5ba4b221dfa" volumeName="kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-stats-auth" seLinuxMountContext=""
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828341    4956 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:22:49.830148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.828351    4956 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:22:49.849316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.849284    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/1.log"
Nov 02 17:22:49.852939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.852912    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/1.log"
Nov 02 17:22:49.854454 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.854206    4956 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:22:49.856780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.856755    4956 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:22:49.856856 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.856786    4956 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:22:49.856856 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.856836    4956 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:22:49.856945 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.856878    4956 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 17:22:49.880524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.880499    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/1.log"
Nov 02 17:22:49.880690 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.880532    4956 generic.go:356] "Generic (PLEG): container finished" podID="eedb70cd-4f51-41ae-acfd-6c5c2044644d" containerID="6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818" exitCode=255
Nov 02 17:22:49.885881 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.885848    4956 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="0491ceacad4e56a5e30e7411e0ed52cb3364817644ae8bbc2426dfb5b6f2eedc" exitCode=0
Nov 02 17:22:49.885881 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.885880    4956 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="090f004b972e77cdc9c095d41a41852a7edb0de5e613d932c83d99d8cd074ef3" exitCode=0
Nov 02 17:22:49.886055 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.885892    4956 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="401db22b1ff1a9752ed9dffa3d71154df0e9682213cd5caf2c0c17a76ff7eea9" exitCode=0
Nov 02 17:22:49.886055 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.885904    4956 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="04b8357358cbc9aae24b2303a1404b2b776f2178786a2c47d3d794b620908c3b" exitCode=0
Nov 02 17:22:49.886055 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.885913    4956 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="6873eeb1df5eac7e0d251db4bfc9a3782d2fdbb9ae19b4ed1c519e6d19a5eff8" exitCode=0
Nov 02 17:22:49.886055 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.885921    4956 generic.go:356] "Generic (PLEG): container finished" podID="f45ecbf9-1078-4ba0-bb04-0684bc7aadd3" containerID="2084fc8f05cb7e14b617068f5e616daba23457ffdab4ca45d30acfe3e377de54" exitCode=0
Nov 02 17:22:49.890855 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.890832    4956 generic.go:356] "Generic (PLEG): container finished" podID="8b5013ef74faf1be704b590476b00f1a" containerID="926e43b948b600892e01bfe80eacda89c7218c459283ad0bf181aa5a89d1a92b" exitCode=0
Nov 02 17:22:49.922488 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.922369    4956 manager.go:324] Recovery completed
Nov 02 17:22:49.922970 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.922949    4956 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-72.us-west-1.compute.internal\" not found"
Nov 02 17:22:49.940713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.940695    4956 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.942162 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.942144    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.942244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.942175    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.942244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.942188    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.944450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944432    4956 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:22:49.944450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944446    4956 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:22:49.944568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944454    4956 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.944568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944463    4956 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:22:49.944640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944588    4956 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 17:22:49.944640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944598    4956 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 17:22:49.944640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944620    4956 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 17:22:49.944640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944629    4956 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 17:22:49.944640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944636    4956 policy_none.go:50] "Start"
Nov 02 17:22:49.944770 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944645    4956 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:22:49.944770 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944653    4956 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:22:49.944826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944797    4956 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 17:22:49.944826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944805    4956 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 17:22:49.944826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.944810    4956 policy_none.go:44] "Start"
Nov 02 17:22:49.945531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.945515    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.945578 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.945549    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.945578 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.945563    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.949581 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.949556    4956 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:22:49.952171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952153    4956 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:22:49.952252 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.952188    4956 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:22:49.952252 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952205    4956 server.go:87] "Starting device plugin registration server"
Nov 02 17:22:49.952468 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952457    4956 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:22:49.952508 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952469    4956 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:22:49.952615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952596    4956 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:22:49.952740 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952704    4956 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:22:49.952806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952750    4956 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:22:49.952806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952758    4956 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:22:49.952890 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.952790    4956 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:22:49.952738156Z","UUID":"9ff6e1e8-4699-4b8c-bd07-5fc81d474486","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:22:49.954617 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.954584    4956 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:22:49.954798 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.954772    4956 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-72.us-west-1.compute.internal\" not found"
Nov 02 17:22:49.955149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.955105    4956 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:22:49.955257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.955180    4956 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:22:49.957091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.957049    4956 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"]
Nov 02 17:22:49.957246 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.957231    4956 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.958789 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.958768    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.958869 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.958800    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.958869 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.958817    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.959183 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.959169    4956 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.959271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.959256    4956 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.960210 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960194    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.960269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960217    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.960269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960231    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.960269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960235    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.960269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960256    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.960269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960265    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.960490 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.960393    4956 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-72.us-west-1.compute.internal\" not found" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:49.960565 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960533    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"7a902dc8a4f9241e56ab9949e623f113","Type":"ContainerStarted","Data":"38d40e12a7ebee8ef6e7d765747294d39d48aba25d9fe9111f9fa1a17dcaa7fa"}
Nov 02 17:22:49.960615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960578    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"7a902dc8a4f9241e56ab9949e623f113","Type":"ContainerStarted","Data":"1695b7f173e8921619217113dbb5abcfb0e657549f491c5b0fa750f6ee350de8"}
Nov 02 17:22:49.960615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960597    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"8b5013ef74faf1be704b590476b00f1a","Type":"ContainerStarted","Data":"36454a7414909d1de154461df347ffcec1ad8c6b186d05678d137d146238191e"}
Nov 02 17:22:49.960715 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960614    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"8b5013ef74faf1be704b590476b00f1a","Type":"ContainerDied","Data":"926e43b948b600892e01bfe80eacda89c7218c459283ad0bf181aa5a89d1a92b"}
Nov 02 17:22:49.960715 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960623    4956 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.960715 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.960635    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal" event={"ID":"8b5013ef74faf1be704b590476b00f1a","Type":"ContainerStarted","Data":"5d62cd179411e14ff4490a3221e521fae6d2fbc620df7a5ba0e7b5282514a7a9"}
Nov 02 17:22:49.961558 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.961540    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.961625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.961565    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.961625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:49.961575    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.961732 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:49.961720    4956 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-72.us-west-1.compute.internal\" not found" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.015607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.015566    4956 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:50.016476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.016458    4956 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:50.016568 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.016466    4956 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 17:22:50.025758 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.025729    4956 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.052899 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.052870    4956 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:50.054173 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.054151    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:50.054261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.054187    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:50.054261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.054198    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:50.054261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.054250    4956 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.066114 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.066086    4956 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:50.066301 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.066285    4956 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:50.085791 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.085764    4956 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:50.085884 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:50.085819    4956 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.085884 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.085833    4956 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.117197 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.117164    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.117343 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.117203    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.117343 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.117228    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a902dc8a4f9241e56ab9949e623f113-config\") pod \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"7a902dc8a4f9241e56ab9949e623f113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.143903 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.143873    4956 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:50.143974 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:50.143929    4956 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.167505 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.167469    4956 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.167683 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.167551    4956 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.168194 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.168177    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeNotReady"
Nov 02 17:22:50.168243 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.168212    4956 setters.go:546] "Node became not ready" node="ip-10-0-129-72.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T17:22:50Z","lastTransitionTime":"2026-11-02T17:22:50Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 02 17:22:50.176375 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.176320    4956 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-72.us-west-1.compute.internal" is owned by different node (oldNodeID="7f5141e6-1eaf-4711-a200-3db99714169d", newNodeID=""), cleaning up...
Nov 02 17:22:50.217533 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.217502    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.217533 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.217534    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.217672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.217552    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a902dc8a4f9241e56ab9949e623f113-config\") pod \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"7a902dc8a4f9241e56ab9949e623f113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.217672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.217603    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.217672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.217618    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b5013ef74faf1be704b590476b00f1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"8b5013ef74faf1be704b590476b00f1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.217672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.217649    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a902dc8a4f9241e56ab9949e623f113-config\") pod \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" (UID: \"7a902dc8a4f9241e56ab9949e623f113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.435648 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.435579    4956 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-72.us-west-1.compute.internal" event="NodeReady"
Nov 02 17:22:50.435784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.435661    4956 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 17:22:50.809338 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.809298    4956 apiserver.go:52] "Watching apiserver"
Nov 02 17:22:50.917666 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.917606    4956 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.918201 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.918184    4956 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.962189 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.962155    4956 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:22:50.962921 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.962890    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-m7rhm","kube-system/kas-connection-checker-685895c7c9-tbpf9","openshift-ingress-canary/ingress-canary-8v2j6","kube-system/kas-connection-checker-685895c7c9-czpdz","openshift-dns/dns-default-rn2gc","openshift-network-diagnostics/network-check-target-przls","openshift-ovn-kubernetes/ovnkube-node-455n9","kube-system/global-pull-secret-syncer-9m4h8","openshift-multus/multus-additional-cni-plugins-g7sgh","openshift-insights/insights-operator-7c4986d449-5hwls","kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-8n69x","openshift-network-operator/iptables-alerter-mj6cq","openshift-image-registry/image-registry-67778b997c-255bm","openshift-dns/node-resolver-zfl47","openshift-ingress/router-default-6d9d94667c-xzxd4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl","kube-system/kas-connection-checker-685895c7c9-gkwvc","openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt","openshift-image-registry/image-registry-76ff76747b-h7b52","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk","openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff","openshift-image-registry/node-ca-gmfdg","openshift-multus/multus-586s4","openshift-network-diagnostics/network-check-source-6f78d59f99-988dm","openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk","openshift-network-console/networking-console-plugin-86c478bb88-7mpvr","openshift-console-operator/console-operator-7cd56b8dbc-zhfgw","openshift-multus/network-metrics-daemon-h7btw","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm","openshift-cluster-node-tuning-operator/tuned-q26xm"]
Nov 02 17:22:50.964806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.964779    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:50.965394 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.965376    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:50.966025 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.966003    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:50.967078 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.967059    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:50.967481 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.967465    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:50.970187 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.970167    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:50.974404 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.974383    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:50.974971 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.974956    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:50.976182 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.976166    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:50.980106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.980086    4956 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:50.980216 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:50.980154    4956 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:50.981223 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:50.981211    4956 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:50.981267 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:50.981249    4956 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-72.us-west-1.compute.internal"
Nov 02 17:22:51.015705 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.015681    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.016410 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.016394    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.021024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021005    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 17:22:51.021024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021011    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 17:22:51.021174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021013    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 17:22:51.021174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021032    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 17:22:51.021590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021546    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 17:22:51.021590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021582    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v649h\""
Nov 02 17:22:51.021725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021585    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 17:22:51.021725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021585    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 17:22:51.021725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021654    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-djvvn\""
Nov 02 17:22:51.021725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021674    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gfr4c\""
Nov 02 17:22:51.021725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021704    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 17:22:51.021725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021715    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5vhr8\""
Nov 02 17:22:51.021930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021788    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vb5h7\""
Nov 02 17:22:51.021930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021846    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-s74tj\""
Nov 02 17:22:51.021930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021912    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lh9zn\""
Nov 02 17:22:51.022014 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.021986    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7t5cr\""
Nov 02 17:22:51.022471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.022452    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 17:22:51.022545 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.022510    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 17:22:51.022545 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.022523    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 17:22:51.022697 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.022678    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 17:22:51.022751 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.022697    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xcdtc\""
Nov 02 17:22:51.022751 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.022731    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fb2bd\""
Nov 02 17:22:51.025371 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.025357    4956 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:22:51.063326 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.063255    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 17:22:51.063433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.063332    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wf5x\""
Nov 02 17:22:51.063433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.063410    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztdmt\""
Nov 02 17:22:51.063518 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.063505    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-nxqk5\""
Nov 02 17:22:51.063574 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.063563    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l2rxq\""
Nov 02 17:22:51.064936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.064922    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:22:51.065027 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065014    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:22:51.065171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065155    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:22:51.065271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065248    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b28rw\""
Nov 02 17:22:51.065271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065267    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:22:51.065431 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065283    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xdlgk\""
Nov 02 17:22:51.065431 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065320    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2zcw\""
Nov 02 17:22:51.065431 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065396    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:22:51.065431 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065425    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nt8\""
Nov 02 17:22:51.065583 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065462    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:22:51.065583 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065478    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4464\""
Nov 02 17:22:51.065583 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065474    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v5b9c\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065593    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7kdpj\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065601    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkkxw\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065616    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065650    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svxv2\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065658    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065661    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-546w9\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065677    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-867zk\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065677    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l2dvj\""
Nov 02 17:22:51.065713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.065699    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jxqfl\""
Nov 02 17:22:51.116104 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116070    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:22:51.116550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116523    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.116550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116523    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116622    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116634    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116685    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116646    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116702    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116625    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116724    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116679    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116744    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.116784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116771    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116812    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116821    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116847    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116871    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116874    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116889    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116932    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116949    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116960    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116966    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.116979    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117092    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117114    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117131    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117141    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117149    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117175    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117176    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117196    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117196    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117217    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117208    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117239    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117256    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.117280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117258    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117302    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117327    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117336    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117344    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117358    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117365    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117404    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117480    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117494    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117508    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117544    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117629    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117638    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117711    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117715    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117745    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:22:51.118340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.117764    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 17:22:51.118867 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.118587    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 17:22:51.121875 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.121858    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-log-socket\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.121926 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.121886    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-stats-auth\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.121926 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.121904    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-kubernetes\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.121926 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.121921    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-tuned\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.122063 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.121940    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.122063 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122009    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5998295-35aa-458b-9174-625f915c16cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:51.122174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122069    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-slash\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.122174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122106    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-tuned\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.122287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122197    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:51.122287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122231    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.122287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122255    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5998295-35aa-458b-9174-625f915c16cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:51.122287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122261    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-stats-auth\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.122287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122284    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.122448 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122309    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cnibin\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.122448 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122355    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-config\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.122448 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122380    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.122448 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122397    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.122448 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122415    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:51.122448 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122444    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122472    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-config-volume\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122497    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-tmp\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122520    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122548    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedb70cd-4f51-41ae-acfd-6c5c2044644d-serving-cert\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122575    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjmg\" (UniqueName: \"kubernetes.io/projected/eedb70cd-4f51-41ae-acfd-6c5c2044644d-kube-api-access-qtjmg\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122612    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-tmp\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122615    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122600    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33553579-055a-4dd9-986d-9e32c86d442f-crio-socket\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122654    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.122700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122659    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-config\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122706    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-config-volume\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122761    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122833    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eedb70cd-4f51-41ae-acfd-6c5c2044644d-serving-cert\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122843    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-system-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122890    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-netns\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122920    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122938    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-var-lib-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.122983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122969    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxgv\" (UniqueName: \"kubernetes.io/projected/f2fc2433-5459-47d9-a32f-511bf10bcf70-kube-api-access-5qxgv\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.123225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.122999    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.123225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123029    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/45fb7f7a-5ad6-4176-8405-ece930a49aca-kube-api-access-flcdx\") pod \"volume-data-source-validator-58c9f5f46d-6c5hm\" (UID: \"45fb7f7a-5ad6-4176-8405-ece930a49aca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"
Nov 02 17:22:51.123225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123055    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.123225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123136    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:51.123225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123171    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs858\" (UniqueName: \"kubernetes.io/projected/8a562913-9258-41b3-960e-c9dc07e2637c-kube-api-access-xs858\") pod \"kas-connection-checker-685895c7c9-gkwvc\" (UID: \"8a562913-9258-41b3-960e-c9dc07e2637c\") " pod="kube-system/kas-connection-checker-685895c7c9-gkwvc"
Nov 02 17:22:51.123225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123190    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.123463 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123248    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-daemon-config\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.123463 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123289    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.123463 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123309    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.123463 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123432    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdw4\" (UniqueName: \"kubernetes.io/projected/a1767839-4632-46c7-84aa-19269b3be129-kube-api-access-lzdw4\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:51.123641 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123481    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64d157fc-a912-44a3-9359-23691282dda2-original-pull-secret\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.123641 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123632    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-daemon-config\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.123782 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123753    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-run\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.123913 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123812    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsr5f\" (UniqueName: \"kubernetes.io/projected/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc-kube-api-access-bsr5f\") pod \"kas-connection-checker-685895c7c9-czpdz\" (UID: \"c8f53a9e-fe98-4056-8d6b-29bd6f1912cc\") " pod="kube-system/kas-connection-checker-685895c7c9-czpdz"
Nov 02 17:22:51.123913 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123840    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-systemd-units\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.123913 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123865    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-ovn\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.123913 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123887    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-lib-modules\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123930    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-hostroot\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.123984    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-device-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124018    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-config\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124043    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cni-binary-copy\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124085    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-kubelet\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124111    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33553579-055a-4dd9-986d-9e32c86d442f-data-volume\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124153    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-conf-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.124203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124177    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-snapshots\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124236    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-config\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124237    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-socket-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124324    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2fc2433-5459-47d9-a32f-511bf10bcf70-snapshots\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124369    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-netns\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124399    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-script-lib\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124424    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvvm\" (UniqueName: \"kubernetes.io/projected/78779265-8cc2-4286-8ad2-3aed34f341dd-kube-api-access-bgvvm\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124449    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cni-binary-copy\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124455    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmrr\" (UniqueName: \"kubernetes.io/projected/d269254c-0c37-4fad-b7e4-20b88ee2c2c1-kube-api-access-5hmrr\") pod \"kas-connection-checker-685895c7c9-tbpf9\" (UID: \"d269254c-0c37-4fad-b7e4-20b88ee2c2c1\") " pod="kube-system/kas-connection-checker-685895c7c9-tbpf9"
Nov 02 17:22:51.124548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124534    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33553579-055a-4dd9-986d-9e32c86d442f-data-volume\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124569    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e170d8-e67b-4972-9e92-28b033c07336-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124595    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e170d8-e67b-4972-9e92-28b033c07336-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124620    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwczd\" (UniqueName: \"kubernetes.io/projected/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-kube-api-access-bwczd\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124720    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a35c68-067f-4fac-82f0-a7ef888b21fb-iptables-alerter-script\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124794    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-ovnkube-script-lib\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124806    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e170d8-e67b-4972-9e92-28b033c07336-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124812    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124846    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e170d8-e67b-4972-9e92-28b033c07336-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124879    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysconfig\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.124918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124902    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124939    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-serving-cert\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.124962    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-tmp-dir\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125050    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-tmp-dir\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125059    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-serving-cert\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125062    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a35c68-067f-4fac-82f0-a7ef888b21fb-iptables-alerter-script\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125057    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52674954-19d1-447d-ade5-d8655b4db351-tmp-dir\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125135    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125146    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52674954-19d1-447d-ade5-d8655b4db351-tmp-dir\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125190    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aae52ad6-e69f-4db7-8924-3927e0b9af50-cert\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125257    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wxdc\" (UniqueName: \"kubernetes.io/projected/cd56f1da-de64-4541-9c26-e5ba4b221dfa-kube-api-access-9wxdc\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.125275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125262    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125330    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-multus\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125396    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aae52ad6-e69f-4db7-8924-3927e0b9af50-cert\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125425    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-kubelet\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125490    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p75t\" (UniqueName: \"kubernetes.io/projected/52674954-19d1-447d-ade5-d8655b4db351-kube-api-access-5p75t\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125536    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8mg\" (UniqueName: \"kubernetes.io/projected/aae52ad6-e69f-4db7-8924-3927e0b9af50-kube-api-access-rm8mg\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125556    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125572    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-netd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.125619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125598    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-sys\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.125925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125625    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dbg\" (UniqueName: \"kubernetes.io/projected/de90b397-c8f3-4b8e-b89a-822b7bc465bd-kube-api-access-79dbg\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.125925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125712    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-konnectivity-ca\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:51.125925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125787    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-systemd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.125925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125801    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:51.125925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125826    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb5d1d52-3223-45e9-a766-848d837c029b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:51.125925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125844    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.125925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125916    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-konnectivity-ca\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125961    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-tmp\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.125992    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-multus-certs\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126017    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-dbus\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126024    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126041    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126047    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb5d1d52-3223-45e9-a766-848d837c029b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126063    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126080    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de90b397-c8f3-4b8e-b89a-822b7bc465bd-tmp\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.126153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126139    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.126418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126170    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrbx\" (UniqueName: \"kubernetes.io/projected/481d442d-4aa0-40bb-b723-ed569f0386ca-kube-api-access-sbrbx\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.126418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126209    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.126418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126277    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cnibin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.126418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126298    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-k8s-cni-cncf-io\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.126418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126342    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.126418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126375    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-node-log\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.126418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126398    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp25t\" (UniqueName: \"kubernetes.io/projected/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-kube-api-access-dp25t\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.126694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126477    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8ec8714-849f-4245-b41e-cb84f8a4996f-host\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:51.126694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126511    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a35c68-067f-4fac-82f0-a7ef888b21fb-host-slash\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:51.126694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126517    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.126694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126539    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndvj\" (UniqueName: \"kubernetes.io/projected/34a35c68-067f-4fac-82f0-a7ef888b21fb-kube-api-access-6ndvj\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:51.126694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126571    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwq8\" (UniqueName: \"kubernetes.io/projected/96e40d68-e813-41fe-8a8f-8c0e3e0cb312-kube-api-access-2cwq8\") pod \"migrator-66df5d496b-gqbmt\" (UID: \"96e40d68-e813-41fe-8a8f-8c0e3e0cb312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"
Nov 02 17:22:51.126694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126631    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/481d442d-4aa0-40bb-b723-ed569f0386ca-ovn-node-metrics-cert\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.126694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126668    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52674954-19d1-447d-ade5-d8655b4db351-hosts-file\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126695    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-sys-fs\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126735    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126757    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/481d442d-4aa0-40bb-b723-ed569f0386ca-ovn-node-metrics-cert\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126762    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskxm\" (UniqueName: \"kubernetes.io/projected/fb5d1d52-3223-45e9-a766-848d837c029b-kube-api-access-gskxm\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126781    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126798    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-env-overrides\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126816    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fc2433-5459-47d9-a32f-511bf10bcf70-serving-cert\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126832    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2d4\" (UniqueName: \"kubernetes.io/projected/d8ec8714-849f-4245-b41e-cb84f8a4996f-kube-api-access-mz2d4\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126856    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-system-cni-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126886    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126913    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-etc-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126937    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-registration-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126957    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126966    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/481d442d-4aa0-40bb-b723-ed569f0386ca-env-overrides\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126983    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-agent-certs\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.126997    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fc2433-5459-47d9-a32f-511bf10bcf70-serving-cert\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.127026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127000    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127035    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127075    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-default-certificate\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127090    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127139    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-systemd\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127196    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5-agent-certs\") pod \"konnectivity-agent-m7rhm\" (UID: \"4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5\") " pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127221    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-etc-kubernetes\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127246    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-trusted-ca\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127251    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127270    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127301    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127330    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxz6p\" (UniqueName: \"kubernetes.io/projected/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-kube-api-access-rxz6p\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127330    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-default-certificate\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127372    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8ec8714-849f-4245-b41e-cb84f8a4996f-serviceca\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127409    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127428    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127437    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-kubelet-config\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127465    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127493    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflzw\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127529    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127544    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8ec8714-849f-4245-b41e-cb84f8a4996f-serviceca\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127571    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-var-lib-kubelet\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127594    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-service-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127629    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127663    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrvd6\" (UniqueName: \"kubernetes.io/projected/538a6445-5d03-43a2-a41c-0c8e89a53553-kube-api-access-hrvd6\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127697    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-bin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127737    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-config\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127755    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-service-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127774    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqw5\" (UniqueName: \"kubernetes.io/projected/c7260359-2132-4442-8388-5cd2455b5700-kube-api-access-mvqw5\") pod \"network-check-source-6f78d59f99-988dm\" (UID: \"c7260359-2132-4442-8388-5cd2455b5700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127804    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-bin\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.127824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127829    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-modprobe-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127850    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-socket-dir-parent\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127893    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127936    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqnp\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127924    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-config\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.127975    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.128035    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslwl\" (UniqueName: \"kubernetes.io/projected/44e170d8-e67b-4972-9e92-28b033c07336-kube-api-access-vslwl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.128067    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-conf\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.128089    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-host\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.128153    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqw4\" (UniqueName: \"kubernetes.io/projected/33553579-055a-4dd9-986d-9e32c86d442f-kube-api-access-9kqw4\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.128686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.128186    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kr6\" (UniqueName: \"kubernetes.io/projected/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-kube-api-access-f7kr6\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:51.167584 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.167551    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 17:22:51.171289 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.168177    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:22:51.171289 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.169327    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 17:22:51.173478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.173451    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.173751 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.173733    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.173806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.173789    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fc2433-5459-47d9-a32f-511bf10bcf70-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.177987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.177965    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eedb70cd-4f51-41ae-acfd-6c5c2044644d-trusted-ca\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.178802 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.178771    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvkj\" (UniqueName: \"kubernetes.io/projected/c0b00385-648c-4505-a9a8-75070661432f-kube-api-access-jxvkj\") pod \"network-check-target-przls\" (UID: \"c0b00385-648c-4505-a9a8-75070661432f\") " pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:51.179402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.178970    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdw4\" (UniqueName: \"kubernetes.io/projected/a1767839-4632-46c7-84aa-19269b3be129-kube-api-access-lzdw4\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:51.179402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.179035    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjmg\" (UniqueName: \"kubernetes.io/projected/eedb70cd-4f51-41ae-acfd-6c5c2044644d-kube-api-access-qtjmg\") pod \"console-operator-7cd56b8dbc-zhfgw\" (UID: \"eedb70cd-4f51-41ae-acfd-6c5c2044644d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:51.179402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.179055    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxgv\" (UniqueName: \"kubernetes.io/projected/f2fc2433-5459-47d9-a32f-511bf10bcf70-kube-api-access-5qxgv\") pod \"insights-operator-7c4986d449-5hwls\" (UID: \"f2fc2433-5459-47d9-a32f-511bf10bcf70\") " pod="openshift-insights/insights-operator-7c4986d449-5hwls"
Nov 02 17:22:51.179402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.179089    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/45fb7f7a-5ad6-4176-8405-ece930a49aca-kube-api-access-flcdx\") pod \"volume-data-source-validator-58c9f5f46d-6c5hm\" (UID: \"45fb7f7a-5ad6-4176-8405-ece930a49aca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6c5hm"
Nov 02 17:22:51.179402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.179351    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmrr\" (UniqueName: \"kubernetes.io/projected/d269254c-0c37-4fad-b7e4-20b88ee2c2c1-kube-api-access-5hmrr\") pod \"kas-connection-checker-685895c7c9-tbpf9\" (UID: \"d269254c-0c37-4fad-b7e4-20b88ee2c2c1\") " pod="kube-system/kas-connection-checker-685895c7c9-tbpf9"
Nov 02 17:22:51.179402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.179376    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.179402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.179384    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs858\" (UniqueName: \"kubernetes.io/projected/8a562913-9258-41b3-960e-c9dc07e2637c-kube-api-access-xs858\") pod \"kas-connection-checker-685895c7c9-gkwvc\" (UID: \"8a562913-9258-41b3-960e-c9dc07e2637c\") " pod="kube-system/kas-connection-checker-685895c7c9-gkwvc"
Nov 02 17:22:51.179637 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.179453    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsr5f\" (UniqueName: \"kubernetes.io/projected/c8f53a9e-fe98-4056-8d6b-29bd6f1912cc-kube-api-access-bsr5f\") pod \"kas-connection-checker-685895c7c9-czpdz\" (UID: \"c8f53a9e-fe98-4056-8d6b-29bd6f1912cc\") " pod="kube-system/kas-connection-checker-685895c7c9-czpdz"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.182905    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwczd\" (UniqueName: \"kubernetes.io/projected/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-kube-api-access-bwczd\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183016    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dbg\" (UniqueName: \"kubernetes.io/projected/de90b397-c8f3-4b8e-b89a-822b7bc465bd-kube-api-access-79dbg\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183025    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndvj\" (UniqueName: \"kubernetes.io/projected/34a35c68-067f-4fac-82f0-a7ef888b21fb-kube-api-access-6ndvj\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183019    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwq8\" (UniqueName: \"kubernetes.io/projected/96e40d68-e813-41fe-8a8f-8c0e3e0cb312-kube-api-access-2cwq8\") pod \"migrator-66df5d496b-gqbmt\" (UID: \"96e40d68-e813-41fe-8a8f-8c0e3e0cb312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gqbmt"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183065    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p75t\" (UniqueName: \"kubernetes.io/projected/52674954-19d1-447d-ade5-d8655b4db351-kube-api-access-5p75t\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183067    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvvm\" (UniqueName: \"kubernetes.io/projected/78779265-8cc2-4286-8ad2-3aed34f341dd-kube-api-access-bgvvm\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183300    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrbx\" (UniqueName: \"kubernetes.io/projected/481d442d-4aa0-40bb-b723-ed569f0386ca-kube-api-access-sbrbx\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.183352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183329    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wxdc\" (UniqueName: \"kubernetes.io/projected/cd56f1da-de64-4541-9c26-e5ba4b221dfa-kube-api-access-9wxdc\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.183617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183439    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.183617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183517    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp25t\" (UniqueName: \"kubernetes.io/projected/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-kube-api-access-dp25t\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.183706 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.183690    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8mg\" (UniqueName: \"kubernetes.io/projected/aae52ad6-e69f-4db7-8924-3927e0b9af50-kube-api-access-rm8mg\") pod \"ingress-canary-8v2j6\" (UID: \"aae52ad6-e69f-4db7-8924-3927e0b9af50\") " pod="openshift-ingress-canary/ingress-canary-8v2j6"
Nov 02 17:22:51.184422 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.184403    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslwl\" (UniqueName: \"kubernetes.io/projected/44e170d8-e67b-4972-9e92-28b033c07336-kube-api-access-vslwl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl\" (UID: \"44e170d8-e67b-4972-9e92-28b033c07336\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl"
Nov 02 17:22:51.184935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.184804    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqw5\" (UniqueName: \"kubernetes.io/projected/c7260359-2132-4442-8388-5cd2455b5700-kube-api-access-mvqw5\") pod \"network-check-source-6f78d59f99-988dm\" (UID: \"c7260359-2132-4442-8388-5cd2455b5700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-988dm"
Nov 02 17:22:51.184935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.184880    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2d4\" (UniqueName: \"kubernetes.io/projected/d8ec8714-849f-4245-b41e-cb84f8a4996f-kube-api-access-mz2d4\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:51.185081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.184945    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskxm\" (UniqueName: \"kubernetes.io/projected/fb5d1d52-3223-45e9-a766-848d837c029b-kube-api-access-gskxm\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:51.185081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.184976    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrvd6\" (UniqueName: \"kubernetes.io/projected/538a6445-5d03-43a2-a41c-0c8e89a53553-kube-api-access-hrvd6\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:51.185230 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.185203    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kr6\" (UniqueName: \"kubernetes.io/projected/2a5438c9-cd21-4ac3-93b6-44579f5b7b4e-kube-api-access-f7kr6\") pod \"service-ca-operator-544d749c8c-wwvpk\" (UID: \"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk"
Nov 02 17:22:51.185336 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.185243    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqw4\" (UniqueName: \"kubernetes.io/projected/33553579-055a-4dd9-986d-9e32c86d442f-kube-api-access-9kqw4\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.185414 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.185395    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxz6p\" (UniqueName: \"kubernetes.io/projected/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-kube-api-access-rxz6p\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.185469 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.185446    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqnp\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.185469 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.185462    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflzw\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.229461 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229429    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-registration-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.229461 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229461    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-systemd\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.229668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229476    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-etc-kubernetes\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.229668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229493    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.229668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229557    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-etc-kubernetes\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.229668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229576    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-registration-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.229668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229575    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-systemd\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.229668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229606    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-kubelet-config\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.229668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229665    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229693    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.229700    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229708    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-var-lib-kubelet\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229725    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229742    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-bin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229767    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-bin\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229781    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-modprobe-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229795    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-socket-dir-parent\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229810    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229819    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229829    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229851    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-conf\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229862    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-var-lib-kubelet\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229908    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-modprobe-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229909    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-socket-dir-parent\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229705    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-kubelet-config\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229912    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-host\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.229713    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229868    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-host\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229950    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-log-socket\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229964    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229967    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-kubernetes\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229995    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230000    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-kubernetes\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229818    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-bin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.229933    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-bin\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230031    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730009347 +0000 UTC m=+1.987039959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:51.230023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230042    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-log-socket\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.229769    4956 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230056    4956 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230099    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-slash\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230139    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730100505 +0000 UTC m=+1.987131125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.229781    4956 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230052    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-slash\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230163    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730149934 +0000 UTC m=+1.987180540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230170    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230205    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730190909 +0000 UTC m=+1.987221507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230219    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230236    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230252    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230270    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cnibin\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230278    4956 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230288    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230291    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-conf\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230305    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730297163 +0000 UTC m=+1.987327802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : secret "metrics-daemon-secret" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230315    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230348    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230358    4956 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230377    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33553579-055a-4dd9-986d-9e32c86d442f-crio-socket\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230384    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730377735 +0000 UTC m=+1.987408381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230403    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-system-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230356    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-cnibin\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230419    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-netns\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230405    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33553579-055a-4dd9-986d-9e32c86d442f-crio-socket\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230428    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysctl-d\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230442    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230452    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-system-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230464    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-netns\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230470    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-var-lib-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230487    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730473977 +0000 UTC m=+1.987504597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230496    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-var-lib-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230498    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230529    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230557    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230588    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-run\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230611    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230613    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-systemd-units\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230623    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230636    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-ovn\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230641    4956 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230661    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-lib-modules\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230672    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730658801 +0000 UTC m=+1.987689400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.230698    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.730689368 +0000 UTC m=+1.987719966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230705    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-ovn\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230714    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-hostroot\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230715    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-run\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230736    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-device-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230745    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-systemd-units\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230755    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-hostroot\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230762    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-lib-modules\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230762    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-kubelet\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230787    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-device-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230790    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-kubelet\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230799    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-conf-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230824    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-socket-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230839    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-netns\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230844    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-conf-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230860    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysconfig\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230878    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-multus\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230894    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-kubelet\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230898    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-socket-dir\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230909    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-netd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230931    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-sys\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230933    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-cni-multus\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230940    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-etc-sysconfig\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230946    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-run-netns\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230957    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-systemd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230960    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-var-lib-kubelet\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230971    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-host-cni-netd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230995    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de90b397-c8f3-4b8e-b89a-822b7bc465bd-sys\") pod \"tuned-q26xm\" (UID: \"de90b397-c8f3-4b8e-b89a-822b7bc465bd\") " pod="openshift-cluster-node-tuning-operator/tuned-q26xm"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230996    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-systemd\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231023    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-multus-certs\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.230996    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-multus-certs\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.230989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231049    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-dbus\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231063    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231080    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231096    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231111    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231143    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cnibin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231168    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-run-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231188    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-k8s-cni-cncf-io\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231184    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64d157fc-a912-44a3-9359-23691282dda2-dbus\") pod \"global-pull-secret-syncer-9m4h8\" (UID: \"64d157fc-a912-44a3-9359-23691282dda2\") " pod="kube-system/global-pull-secret-syncer-9m4h8"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231210    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-multus-cni-dir\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231219    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-cnibin\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231214    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-node-log\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231221    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5-host-run-k8s-cni-cncf-io\") pod \"multus-586s4\" (UID: \"2b519ef8-e177-4fb8-9cb0-0e3ee48b2be5\") " pod="openshift-multus/multus-586s4"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231248    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8ec8714-849f-4245-b41e-cb84f8a4996f-host\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231250    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-node-log\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.231192    4956 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231264    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a35c68-067f-4fac-82f0-a7ef888b21fb-host-slash\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231285    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8ec8714-849f-4245-b41e-cb84f8a4996f-host\") pod \"node-ca-gmfdg\" (UID: \"d8ec8714-849f-4245-b41e-cb84f8a4996f\") " pod="openshift-image-registry/node-ca-gmfdg"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231281    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52674954-19d1-447d-ade5-d8655b4db351-hosts-file\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.231299    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.731285063 +0000 UTC m=+1.988315664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231317    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52674954-19d1-447d-ade5-d8655b4db351-hosts-file\") pod \"node-resolver-zfl47\" (UID: \"52674954-19d1-447d-ade5-d8655b4db351\") " pod="openshift-dns/node-resolver-zfl47"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231324    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a35c68-067f-4fac-82f0-a7ef888b21fb-host-slash\") pod \"iptables-alerter-mj6cq\" (UID: \"34a35c68-067f-4fac-82f0-a7ef888b21fb\") " pod="openshift-network-operator/iptables-alerter-mj6cq"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231337    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-sys-fs\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231392    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-system-cni-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231408    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78779265-8cc2-4286-8ad2-3aed34f341dd-sys-fs\") pod \"aws-ebs-csi-driver-node-5p4ff\" (UID: \"78779265-8cc2-4286-8ad2-3aed34f341dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5p4ff"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231414    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-etc-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231430    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f45ecbf9-1078-4ba0-bb04-0684bc7aadd3-system-cni-dir\") pod \"multus-additional-cni-plugins-g7sgh\" (UID: \"f45ecbf9-1078-4ba0-bb04-0684bc7aadd3\") " pod="openshift-multus/multus-additional-cni-plugins-g7sgh"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231434    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/481d442d-4aa0-40bb-b723-ed569f0386ca-etc-openvswitch\") pod \"ovnkube-node-455n9\" (UID: \"481d442d-4aa0-40bb-b723-ed569f0386ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:51.233363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.231716    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33553579-055a-4dd9-986d-9e32c86d442f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.264788 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.264760    4956 scope.go:122] "RemoveContainer" containerID="6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818"
Nov 02 17:22:51.736873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.736831    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:51.737035 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.736889    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:51.737035 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.736911    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:51.737035 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.736937    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:51.737035 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.736964    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:51.737035 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.736993    4956 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737040    4956 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737048    4956 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737052    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737038701 +0000 UTC m=+2.994069300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737108    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737095968 +0000 UTC m=+2.994126567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737054    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737137    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737149    4956 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737168    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737159234 +0000 UTC m=+2.994189878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737184    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737176771 +0000 UTC m=+2.994207369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737194    4956 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.737150    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:51.737245 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737221    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737207176 +0000 UTC m=+2.994237788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737257    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737243843 +0000 UTC m=+2.994274447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : secret "metrics-daemon-secret" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.737272    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.737297    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.737320    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.737337    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737391    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737399    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737405    4956 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737415    4956 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737423    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737415573 +0000 UTC m=+2.994446171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737433    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737428449 +0000 UTC m=+2.994459047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737454    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.737444663 +0000 UTC m=+2.994475278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:51.737632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.737467    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.73745955 +0000 UTC m=+2.994490148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:51.921838 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.921808    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/2.log"
Nov 02 17:22:51.922265 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.922193    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/1.log"
Nov 02 17:22:51.922265 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.922225    4956 generic.go:356] "Generic (PLEG): container finished" podID="eedb70cd-4f51-41ae-acfd-6c5c2044644d" containerID="0ef80e868280a3c9b2823de49d337c5661f824c37e388a64bf649ba4e18fa2e5" exitCode=255
Nov 02 17:22:51.922349 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.922300    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" event={"ID":"eedb70cd-4f51-41ae-acfd-6c5c2044644d","Type":"ContainerDied","Data":"0ef80e868280a3c9b2823de49d337c5661f824c37e388a64bf649ba4e18fa2e5"}
Nov 02 17:22:51.922402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.922366    4956 scope.go:122] "RemoveContainer" containerID="6215605a99c63fa9f4de11c97f1062a5b643a8e0122764c9be288471f860b818"
Nov 02 17:22:51.922881 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:51.922862    4956 scope.go:122] "RemoveContainer" containerID="0ef80e868280a3c9b2823de49d337c5661f824c37e388a64bf649ba4e18fa2e5"
Nov 02 17:22:51.923065 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:51.923045    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:22:52.747863 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.747830    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:52.747863 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.747867    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.747909    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.747934    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.747958    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.747980    4956 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.747988    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748047    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748026867 +0000 UTC m=+5.005057479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748044    4956 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748084    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748072444 +0000 UTC m=+5.005103059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748089    4956 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.748115    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748147    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748107238 +0000 UTC m=+5.005137837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.748185    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748090    4956 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748217    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748203084 +0000 UTC m=+5.005233685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : secret "metrics-daemon-secret" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748212    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748093    4956 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748234    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748242    4956 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748249    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748232872 +0000 UTC m=+5.005263483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748270    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748262245 +0000 UTC m=+5.005292843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748285    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748276455 +0000 UTC m=+5.005307052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:52.748292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748309    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748299427 +0000 UTC m=+5.005330025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:52.748957 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.748356    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:52.748957 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.748391    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:52.748957 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748454    4956 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:52.748957 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748463    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:52.748957 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748471    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:52.748957 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748484    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748473991 +0000 UTC m=+5.005504603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:52.748957 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:52.748497    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.748490674 +0000 UTC m=+5.005521271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:52.927327 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:52.927297    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/2.log"
Nov 02 17:22:53.064717 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:53.064639    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:54.768498 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768435    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:54.768498 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768497    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768524    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768550    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768579    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768590    4956 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768604    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768657    4956 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768675    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.768653418 +0000 UTC m=+9.025684051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768678    4956 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768691    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768706    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67778b997c-255bm: secret "image-registry-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768674    4956 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768724    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs podName:538a6445-5d03-43a2-a41c-0c8e89a53553 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.768705388 +0000 UTC m=+9.025736001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs") pod "network-metrics-daemon-h7btw" (UID: "538a6445-5d03-43a2-a41c-0c8e89a53553") : secret "metrics-daemon-secret" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768745    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.768738468 +0000 UTC m=+9.025769065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768758    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls podName:a1767839-4632-46c7-84aa-19269b3be129 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.768752885 +0000 UTC m=+9.025783482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vdhk" (UID: "a1767839-4632-46c7-84aa-19269b3be129") : secret "samples-operator-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768769    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.768764626 +0000 UTC m=+9.025795223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768791    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls podName:ff6eac8f-a685-462a-88f8-a6bd54d4d1d0 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.768778535 +0000 UTC m=+9.025809146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls") pod "image-registry-67778b997c-255bm" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0") : secret "image-registry-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768860    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768918    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768946    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.768970    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.768984    4956 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769031    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.76901698 +0000 UTC m=+9.026047592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769041    4956 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769053    4956 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76ff76747b-h7b52: secret "image-registry-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769057    4956 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769070    4956 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:54.769061 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769082    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls podName:f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.769070694 +0000 UTC m=+9.026101499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls") pod "image-registry-76ff76747b-h7b52" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1") : secret "image-registry-tls" not found
Nov 02 17:22:54.769889 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769100    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.769090297 +0000 UTC m=+9.026120924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:54.769889 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.769112    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.769106712 +0000 UTC m=+9.026137310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:54.915523 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.915476    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:54.915860 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:54.915846    4956 scope.go:122] "RemoveContainer" containerID="0ef80e868280a3c9b2823de49d337c5661f824c37e388a64bf649ba4e18fa2e5"
Nov 02 17:22:54.916047 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:54.916029    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:22:55.662179 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:55.662151    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zfl47_52674954-19d1-447d-ade5-d8655b4db351/dns-node-resolver/0.log"
Nov 02 17:22:56.262996 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:56.262969    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gmfdg_d8ec8714-849f-4245-b41e-cb84f8a4996f/node-ca/0.log"
Nov 02 17:22:56.916607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:56.916565    4956 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:22:56.916979 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:56.916964    4956 scope.go:122] "RemoveContainer" containerID="0ef80e868280a3c9b2823de49d337c5661f824c37e388a64bf649ba4e18fa2e5"
Nov 02 17:22:56.917185 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:56.917166    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:22:56.961094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:56.961061    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:22:56.961318 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:56.961305    4956 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:22:57.066829 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:57.066804    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8v2j6_aae52ad6-e69f-4db7-8924-3927e0b9af50/serve-healthcheck-canary/0.log"
Nov 02 17:22:57.462492 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:57.462458    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-gqbmt_96e40d68-e813-41fe-8a8f-8c0e3e0cb312/migrator/0.log"
Nov 02 17:22:57.864542 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:57.864467    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl_44e170d8-e67b-4972-9e92-28b033c07336/kube-storage-version-migrator-operator/0.log"
Nov 02 17:22:58.806615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806578    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806629    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806717    4956 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806749    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806776    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert podName:a5998295-35aa-458b-9174-625f915c16cf nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.806762382 +0000 UTC m=+17.063792996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7mpvr" (UID: "a5998295-35aa-458b-9174-625f915c16cf") : secret "networking-console-plugin-cert" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806801    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806816    4956 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806836    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806865    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls podName:fb5d1d52-3223-45e9-a766-848d837c029b nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.806855177 +0000 UTC m=+17.063885775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ghflt" (UID: "fb5d1d52-3223-45e9-a766-848d837c029b") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806879    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806894    4956 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806898    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806906    4956 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806926    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls podName:33553579-055a-4dd9-986d-9e32c86d442f nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.806915106 +0000 UTC m=+17.063945718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8n69x" (UID: "33553579-055a-4dd9-986d-9e32c86d442f") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806945    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806959    4956 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806970    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls podName:f4627fcb-ebee-4c41-9d5b-ee5c0f61624b nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.806954359 +0000 UTC m=+17.063984977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls") pod "dns-default-rn2gc" (UID: "f4627fcb-ebee-4c41-9d5b-ee5c0f61624b") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.806967    4956 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.806993    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.806983251 +0000 UTC m=+17.064013849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : secret "router-metrics-certs-default" not found
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:22:58.807010    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle podName:cd56f1da-de64-4541-9c26-e5ba4b221dfa nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.806999859 +0000 UTC m=+17.064030483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle") pod "router-default-6d9d94667c-xzxd4" (UID: "cd56f1da-de64-4541-9c26-e5ba4b221dfa") : configmap references non-existent config key: service-ca.crt
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.807043    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:58.807091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.807072    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:58.809411 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.809380    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"image-registry-76ff76747b-h7b52\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") " pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:58.809411 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.809400    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1767839-4632-46c7-84aa-19269b3be129-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vdhk\" (UID: \"a1767839-4632-46c7-84aa-19269b3be129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:58.809603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.809494    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538a6445-5d03-43a2-a41c-0c8e89a53553-metrics-certs\") pod \"network-metrics-daemon-h7btw\" (UID: \"538a6445-5d03-43a2-a41c-0c8e89a53553\") " pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:58.809660 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:58.809624    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"image-registry-67778b997c-255bm\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") " pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:59.065800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.065703    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"
Nov 02 17:22:59.067708 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.067359    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:22:59.070827 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.070799    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7btw"
Nov 02 17:22:59.075478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.075220    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:59.183562 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.183534    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:22:59.500333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.500307    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk"]
Nov 02 17:22:59.534035 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.534014    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:22:59.548980 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.548961    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7btw"]
Nov 02 17:22:59.551397 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:59.551342    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538a6445_5d03_43a2_a41c_0c8e89a53553.slice/crio-e7c95bffe5ef448ba23844c834632fec9ae9c538b5623b490005401931ffbcad WatchSource:0}: Error finding container e7c95bffe5ef448ba23844c834632fec9ae9c538b5623b490005401931ffbcad: Status 404 returned error can't find the container with id e7c95bffe5ef448ba23844c834632fec9ae9c538b5623b490005401931ffbcad
Nov 02 17:22:59.552420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.552337    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76ff76747b-h7b52"]
Nov 02 17:22:59.555291 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:59.555265    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6eac8f_a685_462a_88f8_a6bd54d4d1d0.slice/crio-97e8571aef3a1aeb79242113687aae06e4a0455a200bda559ecb21bc2cc07350 WatchSource:0}: Error finding container 97e8571aef3a1aeb79242113687aae06e4a0455a200bda559ecb21bc2cc07350: Status 404 returned error can't find the container with id 97e8571aef3a1aeb79242113687aae06e4a0455a200bda559ecb21bc2cc07350
Nov 02 17:22:59.555982 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:22:59.555962    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a0ae10_d65b_42d0_a5ff_1086bf0f00d1.slice/crio-691e76680e51b843c3320ec6a0f960b322cf2ba2afaf5bc9f83a27c601c95b50 WatchSource:0}: Error finding container 691e76680e51b843c3320ec6a0f960b322cf2ba2afaf5bc9f83a27c601c95b50: Status 404 returned error can't find the container with id 691e76680e51b843c3320ec6a0f960b322cf2ba2afaf5bc9f83a27c601c95b50
Nov 02 17:22:59.599781 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.599749    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67778b997c-255bm"]
Nov 02 17:22:59.950313 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.950269    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7btw" event={"ID":"538a6445-5d03-43a2-a41c-0c8e89a53553","Type":"ContainerStarted","Data":"e7c95bffe5ef448ba23844c834632fec9ae9c538b5623b490005401931ffbcad"}
Nov 02 17:22:59.951807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.951785    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" event={"ID":"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1","Type":"ContainerStarted","Data":"a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67"}
Nov 02 17:22:59.951884 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.951824    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" event={"ID":"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1","Type":"ContainerStarted","Data":"691e76680e51b843c3320ec6a0f960b322cf2ba2afaf5bc9f83a27c601c95b50"}
Nov 02 17:22:59.951929 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.951878    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:22:59.953527 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.953505    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67778b997c-255bm" event={"ID":"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0","Type":"ContainerStarted","Data":"2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3"}
Nov 02 17:22:59.953627 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.953543    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67778b997c-255bm" event={"ID":"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0","Type":"ContainerStarted","Data":"97e8571aef3a1aeb79242113687aae06e4a0455a200bda559ecb21bc2cc07350"}
Nov 02 17:22:59.954605 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:22:59.954585    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk" event={"ID":"a1767839-4632-46c7-84aa-19269b3be129","Type":"ContainerStarted","Data":"f7102156827ad66fd81bd3d591d8e1e61feab6f4c6990026e7ee03ccd232b936"}
Nov 02 17:23:00.957933 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:00.957889    4956 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:23:01.963506 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:01.963461    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk" event={"ID":"a1767839-4632-46c7-84aa-19269b3be129","Type":"ContainerStarted","Data":"526c580ff7f2dad8d2eff35d5dcc97c36206b89ffed3321c3a90cae89abb3a69"}
Nov 02 17:23:01.964141 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:01.963539    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vdhk" event={"ID":"a1767839-4632-46c7-84aa-19269b3be129","Type":"ContainerStarted","Data":"b07ed077838ae63d5df7241e5950b78eb7c86aaf6a49f98b86ef19c4c1e3aeb4"}
Nov 02 17:23:01.965725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:01.965702    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7btw" event={"ID":"538a6445-5d03-43a2-a41c-0c8e89a53553","Type":"ContainerStarted","Data":"177af37fc9d28b2493339e78e809b9859e9dc30507f9b09d6a97b6b617559331"}
Nov 02 17:23:01.965833 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:01.965741    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7btw" event={"ID":"538a6445-5d03-43a2-a41c-0c8e89a53553","Type":"ContainerStarted","Data":"f6ef3e79275bb7082387fbf9dcf5bd034cbac023846e652b0158f1a8e94e6952"}
Nov 02 17:23:02.858180 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:02.858147    4956 scope.go:122] "RemoveContainer" containerID="0ef80e868280a3c9b2823de49d337c5661f824c37e388a64bf649ba4e18fa2e5"
Nov 02 17:23:02.970272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:02.970248    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:23:02.970712 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:02.970696    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/2.log"
Nov 02 17:23:02.970753 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:02.970728    4956 generic.go:356] "Generic (PLEG): container finished" podID="eedb70cd-4f51-41ae-acfd-6c5c2044644d" containerID="8316b504170ea9794ebd3111dd202d1f1feaf1806110d8a85e045b749afa6628" exitCode=255
Nov 02 17:23:02.970822 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:02.970803    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" event={"ID":"eedb70cd-4f51-41ae-acfd-6c5c2044644d","Type":"ContainerDied","Data":"8316b504170ea9794ebd3111dd202d1f1feaf1806110d8a85e045b749afa6628"}
Nov 02 17:23:02.970871 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:02.970861    4956 scope.go:122] "RemoveContainer" containerID="0ef80e868280a3c9b2823de49d337c5661f824c37e388a64bf649ba4e18fa2e5"
Nov 02 17:23:02.971346 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:02.971329    4956 scope.go:122] "RemoveContainer" containerID="8316b504170ea9794ebd3111dd202d1f1feaf1806110d8a85e045b749afa6628"
Nov 02 17:23:02.971559 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:23:02.971538    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:23:03.516729 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:03.516693    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:23:03.584683 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:03.584650    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m7rhm"
Nov 02 17:23:03.978159 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:03.978111    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:23:04.915854 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:04.915808    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:23:04.916244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:04.916228    4956 scope.go:122] "RemoveContainer" containerID="8316b504170ea9794ebd3111dd202d1f1feaf1806110d8a85e045b749afa6628"
Nov 02 17:23:04.916434 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:23:04.916415    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:23:06.885741 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.885707    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:23:06.886149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.885755    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:23:06.886149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.885877    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:23:06.886149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.885928    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:23:06.886149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.886070    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:06.886149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.886148    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:06.886851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.886819    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd56f1da-de64-4541-9c26-e5ba4b221dfa-service-ca-bundle\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:06.888584 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.888562    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd56f1da-de64-4541-9c26-e5ba4b221dfa-metrics-certs\") pod \"router-default-6d9d94667c-xzxd4\" (UID: \"cd56f1da-de64-4541-9c26-e5ba4b221dfa\") " pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:06.888858 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.888833    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33553579-055a-4dd9-986d-9e32c86d442f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n69x\" (UID: \"33553579-055a-4dd9-986d-9e32c86d442f\") " pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:23:06.888946 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.888914    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb5d1d52-3223-45e9-a766-848d837c029b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ghflt\" (UID: \"fb5d1d52-3223-45e9-a766-848d837c029b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:23:06.889039 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.889019    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4627fcb-ebee-4c41-9d5b-ee5c0f61624b-metrics-tls\") pod \"dns-default-rn2gc\" (UID: \"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b\") " pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:23:06.889204 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.889189    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5998295-35aa-458b-9174-625f915c16cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7mpvr\" (UID: \"a5998295-35aa-458b-9174-625f915c16cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:23:06.917284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.917258    4956 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:23:06.917607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:06.917595    4956 scope.go:122] "RemoveContainer" containerID="8316b504170ea9794ebd3111dd202d1f1feaf1806110d8a85e045b749afa6628"
Nov 02 17:23:06.917788 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:23:06.917770    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zhfgw_openshift-console-operator(eedb70cd-4f51-41ae-acfd-6c5c2044644d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" podUID="eedb70cd-4f51-41ae-acfd-6c5c2044644d"
Nov 02 17:23:07.083424 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.083385    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67778b997c-255bm"]
Nov 02 17:23:07.165283 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.165207    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:07.167287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.167269    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"
Nov 02 17:23:07.167956 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.167943    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"
Nov 02 17:23:07.176142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.176095    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:23:07.177690 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.177667    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8n69x"
Nov 02 17:23:07.649915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.649883    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7mpvr"]
Nov 02 17:23:07.652275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.652253    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt"]
Nov 02 17:23:07.652901 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.652877    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rn2gc"]
Nov 02 17:23:07.653476 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:23:07.653448    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5998295_35aa_458b_9174_625f915c16cf.slice/crio-76c797611f0f7717e85179365c90c50d7afe31b07e975609014a9d79aba948de WatchSource:0}: Error finding container 76c797611f0f7717e85179365c90c50d7afe31b07e975609014a9d79aba948de: Status 404 returned error can't find the container with id 76c797611f0f7717e85179365c90c50d7afe31b07e975609014a9d79aba948de
Nov 02 17:23:07.653574 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.653497    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8n69x"]
Nov 02 17:23:07.699446 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.699420    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d9d94667c-xzxd4"]
Nov 02 17:23:07.702139 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:23:07.702085    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33553579_055a_4dd9_986d_9e32c86d442f.slice/crio-42da31043bf700329872c0433b5bb6403cd845d4981835ff4c64ff4335fdf01f WatchSource:0}: Error finding container 42da31043bf700329872c0433b5bb6403cd845d4981835ff4c64ff4335fdf01f: Status 404 returned error can't find the container with id 42da31043bf700329872c0433b5bb6403cd845d4981835ff4c64ff4335fdf01f
Nov 02 17:23:07.703242 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:23:07.703215    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5d1d52_3223_45e9_a766_848d837c029b.slice/crio-5181b08671c0a73552b4ec38f91da83cd1f59880977ee1bead1c7cd73d2343b2 WatchSource:0}: Error finding container 5181b08671c0a73552b4ec38f91da83cd1f59880977ee1bead1c7cd73d2343b2: Status 404 returned error can't find the container with id 5181b08671c0a73552b4ec38f91da83cd1f59880977ee1bead1c7cd73d2343b2
Nov 02 17:23:07.703580 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:23:07.703558    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd56f1da_de64_4541_9c26_e5ba4b221dfa.slice/crio-c299d68f244c6e833d73873935ec2ffa363f7adae62f5e4affb99fac8ea44b95 WatchSource:0}: Error finding container c299d68f244c6e833d73873935ec2ffa363f7adae62f5e4affb99fac8ea44b95: Status 404 returned error can't find the container with id c299d68f244c6e833d73873935ec2ffa363f7adae62f5e4affb99fac8ea44b95
Nov 02 17:23:07.704413 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:23:07.704395    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4627fcb_ebee_4c41_9d5b_ee5c0f61624b.slice/crio-0bd7afd2ce9786298ff9f650018f48c65d343f7517d2963a89a3b574d65ce55f WatchSource:0}: Error finding container 0bd7afd2ce9786298ff9f650018f48c65d343f7517d2963a89a3b574d65ce55f: Status 404 returned error can't find the container with id 0bd7afd2ce9786298ff9f650018f48c65d343f7517d2963a89a3b574d65ce55f
Nov 02 17:23:07.996196 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.996155    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr" event={"ID":"a5998295-35aa-458b-9174-625f915c16cf","Type":"ContainerStarted","Data":"76c797611f0f7717e85179365c90c50d7afe31b07e975609014a9d79aba948de"}
Nov 02 17:23:07.997385 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.997360    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn2gc" event={"ID":"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b","Type":"ContainerStarted","Data":"0bd7afd2ce9786298ff9f650018f48c65d343f7517d2963a89a3b574d65ce55f"}
Nov 02 17:23:07.998483 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.998461    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt" event={"ID":"fb5d1d52-3223-45e9-a766-848d837c029b","Type":"ContainerStarted","Data":"5181b08671c0a73552b4ec38f91da83cd1f59880977ee1bead1c7cd73d2343b2"}
Nov 02 17:23:07.999923 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.999903    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d9d94667c-xzxd4" event={"ID":"cd56f1da-de64-4541-9c26-e5ba4b221dfa","Type":"ContainerStarted","Data":"5f4205cecafdd141fc29aac679e8124f2d8dd5d9e3abe0fea0183618afb03b1c"}
Nov 02 17:23:08.000035 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:07.999938    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d9d94667c-xzxd4" event={"ID":"cd56f1da-de64-4541-9c26-e5ba4b221dfa","Type":"ContainerStarted","Data":"c299d68f244c6e833d73873935ec2ffa363f7adae62f5e4affb99fac8ea44b95"}
Nov 02 17:23:08.001454 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:08.001434    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n69x" event={"ID":"33553579-055a-4dd9-986d-9e32c86d442f","Type":"ContainerStarted","Data":"12712302bcf5b7275214cb8dca0f47869ec181f2bce678da8d95d09bac95ecf8"}
Nov 02 17:23:08.001526 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:08.001477    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n69x" event={"ID":"33553579-055a-4dd9-986d-9e32c86d442f","Type":"ContainerStarted","Data":"42da31043bf700329872c0433b5bb6403cd845d4981835ff4c64ff4335fdf01f"}
Nov 02 17:23:08.166195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:08.166154    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:08.168946 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:08.168913    4956 patch_prober.go:29] interesting pod/router-default-6d9d94667c-xzxd4 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Nov 02 17:23:08.168946 ip-10-0-129-72 kubenswrapper[4956]: [-]has-synced failed: reason withheld
Nov 02 17:23:08.168946 ip-10-0-129-72 kubenswrapper[4956]: [+]process-running ok
Nov 02 17:23:08.168946 ip-10-0-129-72 kubenswrapper[4956]: healthz check failed
Nov 02 17:23:08.169171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:08.168972    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-6d9d94667c-xzxd4" podUID="cd56f1da-de64-4541-9c26-e5ba4b221dfa" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 02 17:23:09.167675 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:09.167644    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:10.021039 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:10.021007    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7mpvr" event={"ID":"a5998295-35aa-458b-9174-625f915c16cf","Type":"ContainerStarted","Data":"3b2ba1d6963831c8219c2b89e888fca3f1b9d3913ae125e691502b2654fcb11b"}
Nov 02 17:23:10.024758 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:10.024723    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n69x" event={"ID":"33553579-055a-4dd9-986d-9e32c86d442f","Type":"ContainerStarted","Data":"d5151c0e4af8c4d9360f5a0b6b46a35c6b17f6395a36dcc1380881c70d7b3f0a"}
Nov 02 17:23:10.024913 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:10.024791    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:10.026000 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:10.025982    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d9d94667c-xzxd4"
Nov 02 17:23:11.028767 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:11.028730    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn2gc" event={"ID":"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b","Type":"ContainerStarted","Data":"27fed0a8e73425e84e546ff7c8078d9b869aed0ea86f7be9b228d7c092169fec"}
Nov 02 17:23:11.029244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:11.028818    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn2gc" event={"ID":"f4627fcb-ebee-4c41-9d5b-ee5c0f61624b","Type":"ContainerStarted","Data":"868246780da76c40b2420eeea675bd21a4fa103336a65c7f6cabaec27896626b"}
Nov 02 17:23:11.029244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:11.028871    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:23:11.030374 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:11.030348    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ghflt" event={"ID":"fb5d1d52-3223-45e9-a766-848d837c029b","Type":"ContainerStarted","Data":"f18cfb370e44bd9b2f7436050b7c13cea049a59a7ca3168d18725030d3c2e4b2"}
Nov 02 17:23:11.032541 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:11.032518    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n69x" event={"ID":"33553579-055a-4dd9-986d-9e32c86d442f","Type":"ContainerStarted","Data":"3e4faa72583aac7ae4e1121e94da8c6c2bde6f17c640631f62977ea916a18f82"}
Nov 02 17:23:14.893912 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:14.893879    4956 patch_prober.go:29] interesting pod/image-registry-76ff76747b-h7b52 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 17:23:14.894425 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:14.893937    4956 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:23:17.088433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:17.088401    4956 patch_prober.go:29] interesting pod/image-registry-67778b997c-255bm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 17:23:17.088836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:17.088454    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67778b997c-255bm" podUID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:23:19.186233 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:19.186204    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-przls"
Nov 02 17:23:20.962106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:20.962072    4956 patch_prober.go:29] interesting pod/image-registry-76ff76747b-h7b52 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 17:23:20.962534 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:20.962136    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:23:21.038727 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:21.038692    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rn2gc"
Nov 02 17:23:21.077531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:21.077496    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-455n9"
Nov 02 17:23:23.911094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:23.911063    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5x8kk"]
Nov 02 17:23:23.914654 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:23.914635    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.046777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.046745    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-sys\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.046777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.046784    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-tls\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.047010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.046805    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-wtmp\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.047010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.046871    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ad288b-fc96-4d66-9c2c-336ae6e60afb-metrics-client-ca\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.047010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.046917    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-textfile\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.047211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.047067    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.047211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.047142    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqz6v\" (UniqueName: \"kubernetes.io/projected/20ad288b-fc96-4d66-9c2c-336ae6e60afb-kube-api-access-hqz6v\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.047211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.047181    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.047315 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.047264    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-root\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.086475 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.086417    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 17:23:24.086672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.086537    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 17:23:24.086672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.086610    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 17:23:24.086672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.086631    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 17:23:24.086829 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.086686    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hv5xv\""
Nov 02 17:23:24.148032 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.147997    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148032 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148036    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-root\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148066    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-sys\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148088    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-tls\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148159    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-wtmp\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148182    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-sys\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148199    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ad288b-fc96-4d66-9c2c-336ae6e60afb-metrics-client-ca\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148265    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-textfile\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148308    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148703 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148325    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-wtmp\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148703 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148269    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20ad288b-fc96-4d66-9c2c-336ae6e60afb-root\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148703 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148348    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqz6v\" (UniqueName: \"kubernetes.io/projected/20ad288b-fc96-4d66-9c2c-336ae6e60afb-kube-api-access-hqz6v\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148703 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148650    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-textfile\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148797    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20ad288b-fc96-4d66-9c2c-336ae6e60afb-metrics-client-ca\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.148868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.148810    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.151171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.151145    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.151282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.151213    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20ad288b-fc96-4d66-9c2c-336ae6e60afb-node-exporter-tls\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.206060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.206025    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqz6v\" (UniqueName: \"kubernetes.io/projected/20ad288b-fc96-4d66-9c2c-336ae6e60afb-kube-api-access-hqz6v\") pod \"node-exporter-5x8kk\" (UID: \"20ad288b-fc96-4d66-9c2c-336ae6e60afb\") " pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.226882 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.226851    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5x8kk"
Nov 02 17:23:24.237307 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:23:24.237278    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ad288b_fc96_4d66_9c2c_336ae6e60afb.slice/crio-df4a033554e8c3be07615c63013c594214dc2348b856c4b66613a66fc90d4112 WatchSource:0}: Error finding container df4a033554e8c3be07615c63013c594214dc2348b856c4b66613a66fc90d4112: Status 404 returned error can't find the container with id df4a033554e8c3be07615c63013c594214dc2348b856c4b66613a66fc90d4112
Nov 02 17:23:24.893740 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.893707    4956 patch_prober.go:29] interesting pod/image-registry-76ff76747b-h7b52 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 17:23:24.893925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:24.893767    4956 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:23:25.084798 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:25.084756    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5x8kk" event={"ID":"20ad288b-fc96-4d66-9c2c-336ae6e60afb","Type":"ContainerStarted","Data":"df4a033554e8c3be07615c63013c594214dc2348b856c4b66613a66fc90d4112"}
Nov 02 17:23:26.089939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:26.089907    4956 generic.go:356] "Generic (PLEG): container finished" podID="20ad288b-fc96-4d66-9c2c-336ae6e60afb" containerID="3a34a8e3ba1a78a348872dc0433f6984e4ea4ae197ffd69577ee03cc3725cb0d" exitCode=0
Nov 02 17:23:26.090387 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:26.090000    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5x8kk" event={"ID":"20ad288b-fc96-4d66-9c2c-336ae6e60afb","Type":"ContainerDied","Data":"3a34a8e3ba1a78a348872dc0433f6984e4ea4ae197ffd69577ee03cc3725cb0d"}
Nov 02 17:23:27.088049 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:27.088016    4956 patch_prober.go:29] interesting pod/image-registry-67778b997c-255bm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 17:23:27.088238 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:27.088064    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67778b997c-255bm" podUID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:23:27.096130 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:27.096091    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5x8kk" event={"ID":"20ad288b-fc96-4d66-9c2c-336ae6e60afb","Type":"ContainerStarted","Data":"cf248117873c2084c6a86a4d349cd9fc2d8e7f521dd75c5c32fc57d97cfb7c01"}
Nov 02 17:23:27.096464 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:27.096173    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5x8kk" event={"ID":"20ad288b-fc96-4d66-9c2c-336ae6e60afb","Type":"ContainerStarted","Data":"825cff19181fa9d1b9c1f8f9914fd5c6715327408f8000b83e53b9d110cf0488"}
Nov 02 17:23:27.307017 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:27.306960    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5x8kk" podStartSLOduration=2.9231117429999998 podStartE2EDuration="4.306944241s" totalImagesPullingTime="1.305488443s" totalInitContainerRuntime="78.344055ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:27.25743985 +0000 UTC m=+37.514470470" watchObservedRunningTime="2026-11-02 17:23:27.306944241 +0000 UTC m=+37.563974852"
Nov 02 17:23:27.857543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:27.857512    4956 scope.go:122] "RemoveContainer" containerID="8316b504170ea9794ebd3111dd202d1f1feaf1806110d8a85e045b749afa6628"
Nov 02 17:23:28.102318 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:28.102283    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:23:28.102828 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:28.102345    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw" event={"ID":"eedb70cd-4f51-41ae-acfd-6c5c2044644d","Type":"ContainerStarted","Data":"02586ac59f55d731f5506b58afd10cbfaf86d14f7ac4a3c13c45a782997983e3"}
Nov 02 17:23:29.052604 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:23:29.052569    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:23:29.767679 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:29.767599    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76ff76747b-h7b52"]
Nov 02 17:23:29.776510 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:29.776482    4956 patch_prober.go:29] interesting pod/image-registry-76ff76747b-h7b52 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 17:23:29.776647 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:29.776528    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:23:32.103172 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.103106    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67778b997c-255bm" podUID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" containerName="registry" containerID="cri-o://2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3" gracePeriod=30
Nov 02 17:23:32.337768 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.337735    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:23:32.411821 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.411736    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.411821 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.411780    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.411821 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.411823    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.412094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.411848    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.412094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.411878    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.412094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.411944    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.412094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.411983    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.412094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.412058    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\") pod \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\" (UID: \"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0\") "
Nov 02 17:23:32.412573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.412540    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:23:32.412704 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.412641    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:23:32.415172 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.415111    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:23:32.415172 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.415152    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "kube-api-access-mflzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:23:32.415317 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.415231    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:23:32.415317 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.415297    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:23:32.415460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.415440    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:23:32.418897 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.418876    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted" pod "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" (UID: "ff6eac8f-a685-462a-88f8-a6bd54d4d1d0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:23:32.512762 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512724    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mflzw\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-kube-api-access-mflzw\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:32.512762 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512755    4956 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-installation-pull-secrets\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:32.512967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512770    4956 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-image-registry-private-configuration\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:32.512967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512785    4956 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-trusted-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:32.512967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512797    4956 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-certificates\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:32.512967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512809    4956 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-ca-trust-extracted\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:32.512967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512821    4956 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-bound-sa-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:32.512967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:32.512832    4956 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0-registry-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:33.124797 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.124752    4956 generic.go:356] "Generic (PLEG): container finished" podID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" containerID="2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3" exitCode=0
Nov 02 17:23:33.125264 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.124814    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67778b997c-255bm"
Nov 02 17:23:33.125264 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.124837    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67778b997c-255bm" event={"ID":"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0","Type":"ContainerDied","Data":"2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3"}
Nov 02 17:23:33.125264 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.124945    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67778b997c-255bm" event={"ID":"ff6eac8f-a685-462a-88f8-a6bd54d4d1d0","Type":"ContainerDied","Data":"97e8571aef3a1aeb79242113687aae06e4a0455a200bda559ecb21bc2cc07350"}
Nov 02 17:23:33.125264 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.124976    4956 scope.go:122] "RemoveContainer" containerID="2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3"
Nov 02 17:23:33.135850 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.135827    4956 scope.go:122] "RemoveContainer" containerID="2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3"
Nov 02 17:23:33.136178 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:23:33.136153    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3\": container with ID starting with 2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3 not found: ID does not exist" containerID="2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3"
Nov 02 17:23:33.136275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.136185    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3"} err="failed to get container status \"2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3\": rpc error: code = NotFound desc = could not find container \"2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3\": container with ID starting with 2d0de574df544212c74b116fb19522028d430bca9c0d421a4b314fe68e7334a3 not found: ID does not exist"
Nov 02 17:23:33.341002 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.340972    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67778b997c-255bm"]
Nov 02 17:23:33.347855 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.347833    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67778b997c-255bm"]
Nov 02 17:23:33.862797 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:33.862764    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" path="/var/lib/kubelet/pods/ff6eac8f-a685-462a-88f8-a6bd54d4d1d0/volumes"
Nov 02 17:23:38.103394 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:38.103361    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:23:38.107375 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:38.107354    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zhfgw"
Nov 02 17:23:39.772631 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:39.772601    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:23:54.789579 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:54.789516    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry" containerID="cri-o://a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67" gracePeriod=30
Nov 02 17:23:55.037959 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.037935    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:23:55.103368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103279    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.103368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103338    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.103695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103375    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.103695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103405    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.103695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103438    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.103695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103529    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.103695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103571    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.103695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103613    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\") pod \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\" (UID: \"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1\") "
Nov 02 17:23:55.104002 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.103853    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:23:55.104271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.104169    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:23:55.106111 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.106075    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "kube-api-access-cwqnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:23:55.106111 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.106094    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:23:55.106368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.106348    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:23:55.106560 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.106534    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:23:55.106617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.106569    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:23:55.110743 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.110673    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted" pod "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" (UID: "f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:23:55.204589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204555    4956 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-trusted-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.204589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204585    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqnp\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-kube-api-access-cwqnp\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.204589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204596    4956 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-installation-pull-secrets\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.204886 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204606    4956 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-image-registry-private-configuration\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.204886 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204616    4956 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-bound-sa-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.204886 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204624    4956 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-certificates\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.204886 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204633    4956 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-ca-trust-extracted\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.204886 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.204642    4956 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1-registry-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:23:55.213971 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.213941    4956 generic.go:356] "Generic (PLEG): container finished" podID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerID="a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67" exitCode=0
Nov 02 17:23:55.214136 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.213994    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76ff76747b-h7b52"
Nov 02 17:23:55.214136 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.214021    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" event={"ID":"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1","Type":"ContainerDied","Data":"a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67"}
Nov 02 17:23:55.214136 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.214072    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76ff76747b-h7b52" event={"ID":"f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1","Type":"ContainerDied","Data":"691e76680e51b843c3320ec6a0f960b322cf2ba2afaf5bc9f83a27c601c95b50"}
Nov 02 17:23:55.214136 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.214092    4956 scope.go:122] "RemoveContainer" containerID="a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67"
Nov 02 17:23:55.222556 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.222537    4956 scope.go:122] "RemoveContainer" containerID="a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67"
Nov 02 17:23:55.222858 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:23:55.222836    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67\": container with ID starting with a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67 not found: ID does not exist" containerID="a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67"
Nov 02 17:23:55.222912 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.222872    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67"} err="failed to get container status \"a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67\": rpc error: code = NotFound desc = could not find container \"a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67\": container with ID starting with a6d201d77315878c237429a305f7c5fa5c99e1d3731d2d0c340a62f1bea0ae67 not found: ID does not exist"
Nov 02 17:23:55.378107 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.378036    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76ff76747b-h7b52"]
Nov 02 17:23:55.381579 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.381556    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76ff76747b-h7b52"]
Nov 02 17:23:55.862820 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:23:55.862789    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" path="/var/lib/kubelet/pods/f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1/volumes"
Nov 02 17:23:59.051782 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:23:59.051737    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:24:01.238891 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:01.238856    4956 generic.go:356] "Generic (PLEG): container finished" podID="2a5438c9-cd21-4ac3-93b6-44579f5b7b4e" containerID="69905e2b52aab14abb6fc4f36dfc799525e40f0c632337e14304ac89d88fd5d5" exitCode=0
Nov 02 17:24:01.239418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:01.238940    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk" event={"ID":"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e","Type":"ContainerDied","Data":"69905e2b52aab14abb6fc4f36dfc799525e40f0c632337e14304ac89d88fd5d5"}
Nov 02 17:24:01.239418 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:01.239365    4956 scope.go:122] "RemoveContainer" containerID="69905e2b52aab14abb6fc4f36dfc799525e40f0c632337e14304ac89d88fd5d5"
Nov 02 17:24:02.245091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:02.245059    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwvpk" event={"ID":"2a5438c9-cd21-4ac3-93b6-44579f5b7b4e","Type":"ContainerStarted","Data":"72344ffb7cdbb97b910d0960192f875fcc0cc6df0cb62c5ac5ac8ea879445ecd"}
Nov 02 17:24:10.272930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:10.272897    4956 generic.go:356] "Generic (PLEG): container finished" podID="f2fc2433-5459-47d9-a32f-511bf10bcf70" containerID="e0ea498a5796cbd2452153523eefc80504d38e47c8d02c85227838c863027cb6" exitCode=0
Nov 02 17:24:10.273442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:10.272977    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-5hwls" event={"ID":"f2fc2433-5459-47d9-a32f-511bf10bcf70","Type":"ContainerDied","Data":"e0ea498a5796cbd2452153523eefc80504d38e47c8d02c85227838c863027cb6"}
Nov 02 17:24:10.273442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:10.273375    4956 scope.go:122] "RemoveContainer" containerID="e0ea498a5796cbd2452153523eefc80504d38e47c8d02c85227838c863027cb6"
Nov 02 17:24:11.278621 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:11.278583    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-5hwls" event={"ID":"f2fc2433-5459-47d9-a32f-511bf10bcf70","Type":"ContainerStarted","Data":"6976562bcec0f9aafb443af9e8c9b3a60809b95b0f8f7b854ee8245528b95164"}
Nov 02 17:24:25.328965 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:25.328932    4956 generic.go:356] "Generic (PLEG): container finished" podID="44e170d8-e67b-4972-9e92-28b033c07336" containerID="cadb757f728e784e75d258be9ab60da79dd75b924a0c389bc28e3285edce2a87" exitCode=0
Nov 02 17:24:25.329420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:25.328994    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl" event={"ID":"44e170d8-e67b-4972-9e92-28b033c07336","Type":"ContainerDied","Data":"cadb757f728e784e75d258be9ab60da79dd75b924a0c389bc28e3285edce2a87"}
Nov 02 17:24:25.329420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:25.329327    4956 scope.go:122] "RemoveContainer" containerID="cadb757f728e784e75d258be9ab60da79dd75b924a0c389bc28e3285edce2a87"
Nov 02 17:24:26.334461 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:24:26.334419    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-b5fgl" event={"ID":"44e170d8-e67b-4972-9e92-28b033c07336","Type":"ContainerStarted","Data":"40e4b1d8077f28c66fba350df05b748048c910f3937bde5111293df8f4e970ba"}
Nov 02 17:24:29.052411 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:24:29.052372    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:24:59.051930 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:24:59.051892    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:25:29.051665 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:25:29.051624    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:25:59.051786 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:25:59.051750    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:26:29.051545 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:26:29.051504    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:26:39.087772 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.087693    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"]
Nov 02 17:26:39.088232 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.087985    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" containerName="registry"
Nov 02 17:26:39.088232 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.087995    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" containerName="registry"
Nov 02 17:26:39.088232 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.088009    4956 cpu_manager.go:414] "containerMap: removing container" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry"
Nov 02 17:26:39.088232 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.088014    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry"
Nov 02 17:26:39.088232 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.088065    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6eac8f-a685-462a-88f8-a6bd54d4d1d0" containerName="registry"
Nov 02 17:26:39.088232 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.088074    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a0ae10-d65b-42d0-a5ff-1086bf0f00d1" containerName="registry"
Nov 02 17:26:39.090656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.090639    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.153499 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.153467    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 17:26:39.153499 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.153492    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 17:26:39.153704 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.153497    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 17:26:39.153704 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.153510    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 17:26:39.158086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.158048    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.158086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.158096    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nn4t\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-kube-api-access-9nn4t\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.158086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.158228    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.158086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.158275    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc3939-4b5d-41f7-87be-0a3335c95d09-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.158086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.158301    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.158086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.158427    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.158086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.158461    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.200376 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.200338    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gcxmh\""
Nov 02 17:26:39.201354 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.201333    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"]
Nov 02 17:26:39.259839 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.259799    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc3939-4b5d-41f7-87be-0a3335c95d09-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.260024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.259845    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.260024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.259880    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.260024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.259897    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.260024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.259930    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.260024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.259981    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nn4t\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-kube-api-access-9nn4t\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.260024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.260025    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.260749 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.260647    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.262408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.262384    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc3939-4b5d-41f7-87be-0a3335c95d09-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.262510 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.262408    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.262721 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.262705    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.262850 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.262831    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.323182 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.323148    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.323428 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.323411    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nn4t\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-kube-api-access-9nn4t\") pod \"istiod-openshift-gateway-7876bfcb74-txsvx\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.401411 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.401326    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:39.821170 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:39.821141    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"]
Nov 02 17:26:39.824198 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:26:39.824165    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cc3939_4b5d_41f7_87be_0a3335c95d09.slice/crio-41f1c38ba946154387bfa7596f74b0fa72af77d41ec521a50ff676f7f5b0a9dd WatchSource:0}: Error finding container 41f1c38ba946154387bfa7596f74b0fa72af77d41ec521a50ff676f7f5b0a9dd: Status 404 returned error can't find the container with id 41f1c38ba946154387bfa7596f74b0fa72af77d41ec521a50ff676f7f5b0a9dd
Nov 02 17:26:40.811368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:40.811327    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx" event={"ID":"22cc3939-4b5d-41f7-87be-0a3335c95d09","Type":"ContainerStarted","Data":"41f1c38ba946154387bfa7596f74b0fa72af77d41ec521a50ff676f7f5b0a9dd"}
Nov 02 17:26:43.426840 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:43.426806    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:26:43.427259 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:43.426880    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:26:43.824586 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:43.824551    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx" event={"ID":"22cc3939-4b5d-41f7-87be-0a3335c95d09","Type":"ContainerStarted","Data":"40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed"}
Nov 02 17:26:43.824781 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:43.824672    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:44.040630 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:44.040580    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx" podStartSLOduration=1.440498616 podStartE2EDuration="5.040565418s" totalImagesPullingTime="3.600066802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:26:43.990431964 +0000 UTC m=+234.247462578" watchObservedRunningTime="2026-11-02 17:26:44.040565418 +0000 UTC m=+234.297596033"
Nov 02 17:26:44.829968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:44.829939    4956 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-txsvx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:26:44.830478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:44.829993    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx" podUID="22cc3939-4b5d-41f7-87be-0a3335c95d09" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:26:47.829589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:26:47.829550    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:26:59.051478 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:26:59.051437    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:27:29.051269 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:27:29.051231    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:27:39.785243 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.785208    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"]
Nov 02 17:27:39.788635 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.788613    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.834975 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.834927    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.834975 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.834981    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.835268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.835047    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.835268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.835070    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fceb022-1f01-4d2c-897c-afc37313fa99-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.835268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.835091    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6fceb022-1f01-4d2c-897c-afc37313fa99-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.835268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.835205    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvcj\" (UniqueName: \"kubernetes.io/projected/6fceb022-1f01-4d2c-897c-afc37313fa99-kube-api-access-lcvcj\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.917042 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.917001    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 17:27:39.917335 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.917315    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 17:27:39.917441 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.917426    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-z8nm9\""
Nov 02 17:27:39.917557 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.917545    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 17:27:39.917594 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.917544    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 17:27:39.917661 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.917649    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 17:27:39.935781 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.935743    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.935949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.935785    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.935949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.935821    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.935949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.935884    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fceb022-1f01-4d2c-897c-afc37313fa99-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.935949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.935934    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6fceb022-1f01-4d2c-897c-afc37313fa99-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.936188 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.936000    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvcj\" (UniqueName: \"kubernetes.io/projected/6fceb022-1f01-4d2c-897c-afc37313fa99-kube-api-access-lcvcj\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.936399 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.936378    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6fceb022-1f01-4d2c-897c-afc37313fa99-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.938475 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.938447    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fceb022-1f01-4d2c-897c-afc37313fa99-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.938640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.938619    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.938802 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.938776    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:39.938907 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:39.938877    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6fceb022-1f01-4d2c-897c-afc37313fa99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:40.003860 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:40.003831    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvcj\" (UniqueName: \"kubernetes.io/projected/6fceb022-1f01-4d2c-897c-afc37313fa99-kube-api-access-lcvcj\") pod \"opendatahub-operator-controller-manager-89b98757-6624x\" (UID: \"6fceb022-1f01-4d2c-897c-afc37313fa99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:40.052214 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:40.052141    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"]
Nov 02 17:27:40.103532 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:40.103493    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:40.590105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:40.590080    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"]
Nov 02 17:27:40.592533 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:27:40.592506    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fceb022_1f01_4d2c_897c_afc37313fa99.slice/crio-1129cc7465e4f51e41dcdfde52a63a7b5838ae100e6f6cc7a1efbf4e551a3c7f WatchSource:0}: Error finding container 1129cc7465e4f51e41dcdfde52a63a7b5838ae100e6f6cc7a1efbf4e551a3c7f: Status 404 returned error can't find the container with id 1129cc7465e4f51e41dcdfde52a63a7b5838ae100e6f6cc7a1efbf4e551a3c7f
Nov 02 17:27:41.033402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:41.033368    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x" event={"ID":"6fceb022-1f01-4d2c-897c-afc37313fa99","Type":"ContainerStarted","Data":"1129cc7465e4f51e41dcdfde52a63a7b5838ae100e6f6cc7a1efbf4e551a3c7f"}
Nov 02 17:27:44.048765 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:44.048734    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x" event={"ID":"6fceb022-1f01-4d2c-897c-afc37313fa99","Type":"ContainerStarted","Data":"8f3f3d52ab24361b8545ef06fd2094766b2f975c1397e19e5897699c6edee5fc"}
Nov 02 17:27:45.054297 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:45.054264    4956 generic.go:356] "Generic (PLEG): container finished" podID="6fceb022-1f01-4d2c-897c-afc37313fa99" containerID="8f3f3d52ab24361b8545ef06fd2094766b2f975c1397e19e5897699c6edee5fc" exitCode=0
Nov 02 17:27:45.054702 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:45.054308    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x" event={"ID":"6fceb022-1f01-4d2c-897c-afc37313fa99","Type":"ContainerDied","Data":"8f3f3d52ab24361b8545ef06fd2094766b2f975c1397e19e5897699c6edee5fc"}
Nov 02 17:27:46.060158 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:46.060102    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x" event={"ID":"6fceb022-1f01-4d2c-897c-afc37313fa99","Type":"ContainerStarted","Data":"efad7cadc3dad255e39bad06c84418f4f93afb92350b862e3b8ee8c1554ba711"}
Nov 02 17:27:46.060652 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:46.060197    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:46.305730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:46.305677    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x" podStartSLOduration=2.553629635 podStartE2EDuration="7.305662728s" totalImagesPullingTime="4.252278188s" totalInitContainerRuntime="499.754905ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:46.255464587 +0000 UTC m=+296.512495231" watchObservedRunningTime="2026-11-02 17:27:46.305662728 +0000 UTC m=+296.562693348"
Nov 02 17:27:49.882717 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:49.882688    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:27:49.886027 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:49.886008    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:27:57.066332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:27:57.066300    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-6624x"
Nov 02 17:27:59.052302 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:27:59.052269    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:28:20.866556 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.866521    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"]
Nov 02 17:28:20.870151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.870135    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.884650 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.884622    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdld4\" (UniqueName: \"kubernetes.io/projected/0a4c16e4-d81e-4d32-a714-452dd0a06f85-kube-api-access-jdld4\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.884778 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.884657    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4c16e4-d81e-4d32-a714-452dd0a06f85-tls-certs\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.884778 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.884686    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a4c16e4-d81e-4d32-a714-452dd0a06f85-tmp\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.927847 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.927817    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"]
Nov 02 17:28:20.975846 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.975811    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 17:28:20.976069 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.975877    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g78m2\""
Nov 02 17:28:20.976227 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.975876    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 17:28:20.985371 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.985342    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdld4\" (UniqueName: \"kubernetes.io/projected/0a4c16e4-d81e-4d32-a714-452dd0a06f85-kube-api-access-jdld4\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.985484 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.985392    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4c16e4-d81e-4d32-a714-452dd0a06f85-tls-certs\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.985484 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.985443    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a4c16e4-d81e-4d32-a714-452dd0a06f85-tmp\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.987914 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.987891    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a4c16e4-d81e-4d32-a714-452dd0a06f85-tmp\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:20.988208 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:20.988186    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4c16e4-d81e-4d32-a714-452dd0a06f85-tls-certs\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:21.042875 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:21.042847    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdld4\" (UniqueName: \"kubernetes.io/projected/0a4c16e4-d81e-4d32-a714-452dd0a06f85-kube-api-access-jdld4\") pod \"kube-auth-proxy-5d87ddd77c-twqmz\" (UID: \"0a4c16e4-d81e-4d32-a714-452dd0a06f85\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:21.182332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:21.182251    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"
Nov 02 17:28:21.597054 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:21.597012    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz"]
Nov 02 17:28:21.602351 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:28:21.602324    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4c16e4_d81e_4d32_a714_452dd0a06f85.slice/crio-3a241cac0bd8e9f25c73ea3a256d0c697ffa43d67074dea47faa95451b93ce33 WatchSource:0}: Error finding container 3a241cac0bd8e9f25c73ea3a256d0c697ffa43d67074dea47faa95451b93ce33: Status 404 returned error can't find the container with id 3a241cac0bd8e9f25c73ea3a256d0c697ffa43d67074dea47faa95451b93ce33
Nov 02 17:28:21.604144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:21.604096    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:28:22.202724 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:22.202689    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz" event={"ID":"0a4c16e4-d81e-4d32-a714-452dd0a06f85","Type":"ContainerStarted","Data":"3a241cac0bd8e9f25c73ea3a256d0c697ffa43d67074dea47faa95451b93ce33"}
Nov 02 17:28:25.219926 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:25.219884    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz" event={"ID":"0a4c16e4-d81e-4d32-a714-452dd0a06f85","Type":"ContainerStarted","Data":"cf29ae6f2a2f1ef99156b7ba9f81b356698a5accd42c32eb763d9161f81f43aa"}
Nov 02 17:28:25.340483 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:25.340428    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-twqmz" podStartSLOduration=2.400910011 podStartE2EDuration="5.340413242s" totalImagesPullingTime="2.939503231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:25.339340527 +0000 UTC m=+335.596371146" watchObservedRunningTime="2026-11-02 17:28:25.340413242 +0000 UTC m=+335.597443856"
Nov 02 17:28:29.051703 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:28:29.051668    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:28:33.648297 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.648260    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"]
Nov 02 17:28:33.651975 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.651953    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.694642 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.694611    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.694642 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.694655    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.694859 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.694795    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.694859 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.694835    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wct46\" (UniqueName: \"kubernetes.io/projected/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-kube-api-access-wct46\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.709468 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.709436    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-nqs8k\""
Nov 02 17:28:33.759451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.759423    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 17:28:33.759631 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.759553    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 17:28:33.760649 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.760631    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"]
Nov 02 17:28:33.795485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.795450    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.795660 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.795492    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wct46\" (UniqueName: \"kubernetes.io/projected/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-kube-api-access-wct46\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.795660 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.795519    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.795660 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.795558    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.795898 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.795880    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.795953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.795934    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.861949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.861913    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 17:28:33.864613 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.864581    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wct46\" (UniqueName: \"kubernetes.io/projected/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-kube-api-access-wct46\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.866825 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.866795    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e85ee5b1-3fcf-4372-b108-0c7ec63511fa-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fvpqg\" (UID: \"e85ee5b1-3fcf-4372-b108-0c7ec63511fa\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:33.964014 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:33.963974    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:34.386220 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:34.386144    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"]
Nov 02 17:28:34.426591 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:28:34.426556    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85ee5b1_3fcf_4372_b108_0c7ec63511fa.slice/crio-6a3f8c34b2f4b8c244f48a354f4828ff188f46acddf39bef079eb2b4ad3f7a5a WatchSource:0}: Error finding container 6a3f8c34b2f4b8c244f48a354f4828ff188f46acddf39bef079eb2b4ad3f7a5a: Status 404 returned error can't find the container with id 6a3f8c34b2f4b8c244f48a354f4828ff188f46acddf39bef079eb2b4ad3f7a5a
Nov 02 17:28:35.260811 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:35.260779    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg" event={"ID":"e85ee5b1-3fcf-4372-b108-0c7ec63511fa","Type":"ContainerStarted","Data":"6a3f8c34b2f4b8c244f48a354f4828ff188f46acddf39bef079eb2b4ad3f7a5a"}
Nov 02 17:28:37.271197 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:37.271157    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg" event={"ID":"e85ee5b1-3fcf-4372-b108-0c7ec63511fa","Type":"ContainerStarted","Data":"13f722593a6465736cbba8cfcad5f233382a8882adf75d016db6f541d0623da6"}
Nov 02 17:28:37.271632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:37.271259    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:37.436722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:37.436669    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg" podStartSLOduration=2.282429182 podStartE2EDuration="4.436654015s" totalImagesPullingTime="2.154224833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:37.43556977 +0000 UTC m=+347.692600390" watchObservedRunningTime="2026-11-02 17:28:37.436654015 +0000 UTC m=+347.693684636"
Nov 02 17:28:39.364589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.364555    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"]
Nov 02 17:28:39.368048 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.368019    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.445831 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.445791    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba540af9-c669-4cea-9430-7cccf7d46975-manager-config\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.446001 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.445858    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba540af9-c669-4cea-9430-7cccf7d46975-metrics-certs\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.446001 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.445947    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba540af9-c669-4cea-9430-7cccf7d46975-cert\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.446074 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.446004    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96dz\" (UniqueName: \"kubernetes.io/projected/ba540af9-c669-4cea-9430-7cccf7d46975-kube-api-access-z96dz\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.471524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.471479    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 17:28:39.471701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.471593    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 17:28:39.471701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.471614    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:28:39.471701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.471593    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 17:28:39.471701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.471633    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 17:28:39.471845 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.471817    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-flwxn\""
Nov 02 17:28:39.472936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.472919    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"]
Nov 02 17:28:39.546733 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.546698    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba540af9-c669-4cea-9430-7cccf7d46975-metrics-certs\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.546936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.546754    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba540af9-c669-4cea-9430-7cccf7d46975-cert\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.546936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.546820    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96dz\" (UniqueName: \"kubernetes.io/projected/ba540af9-c669-4cea-9430-7cccf7d46975-kube-api-access-z96dz\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.546936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.546848    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba540af9-c669-4cea-9430-7cccf7d46975-manager-config\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.547972 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.547949    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba540af9-c669-4cea-9430-7cccf7d46975-manager-config\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.549641 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.549622    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba540af9-c669-4cea-9430-7cccf7d46975-metrics-certs\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.549795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.549768    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba540af9-c669-4cea-9430-7cccf7d46975-cert\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.604996 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.604956    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96dz\" (UniqueName: \"kubernetes.io/projected/ba540af9-c669-4cea-9430-7cccf7d46975-kube-api-access-z96dz\") pod \"jobset-controller-manager-64b9fccf5-5p8l8\" (UID: \"ba540af9-c669-4cea-9430-7cccf7d46975\") " pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:39.679789 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:39.679704    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:40.050724 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:40.050690    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"]
Nov 02 17:28:40.054470 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:28:40.054445    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba540af9_c669_4cea_9430_7cccf7d46975.slice/crio-d0c6a95ac575c527f480619a04d2c2a13da2818171504b0ef979d3706292c7c6 WatchSource:0}: Error finding container d0c6a95ac575c527f480619a04d2c2a13da2818171504b0ef979d3706292c7c6: Status 404 returned error can't find the container with id d0c6a95ac575c527f480619a04d2c2a13da2818171504b0ef979d3706292c7c6
Nov 02 17:28:40.287064 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:40.287028    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8" event={"ID":"ba540af9-c669-4cea-9430-7cccf7d46975","Type":"ContainerStarted","Data":"d0c6a95ac575c527f480619a04d2c2a13da2818171504b0ef979d3706292c7c6"}
Nov 02 17:28:45.309472 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:45.309434    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8" event={"ID":"ba540af9-c669-4cea-9430-7cccf7d46975","Type":"ContainerStarted","Data":"e195f81c172e6557b596829a9736d541254672c75dbf5f8574b2e8bc3a7b02d8"}
Nov 02 17:28:45.309904 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:45.309542    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:45.430814 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:45.430761    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8" podStartSLOduration=2.214502273 podStartE2EDuration="6.430741866s" totalImagesPullingTime="4.216239593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:45.430017192 +0000 UTC m=+355.687047806" watchObservedRunningTime="2026-11-02 17:28:45.430741866 +0000 UTC m=+355.687772488"
Nov 02 17:28:56.318676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:56.318641    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64b9fccf5-5p8l8"
Nov 02 17:28:58.279824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:28:58.279789    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fvpqg"
Nov 02 17:28:59.051095 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:28:59.051060    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:29:29.052223 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:29:29.052179    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:29:41.976675 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:41.976591    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"]
Nov 02 17:29:41.977156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:41.976851    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx" podUID="22cc3939-4b5d-41f7-87be-0a3335c95d09" containerName="discovery" containerID="cri-o://40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed" gracePeriod=30
Nov 02 17:29:42.222441 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.222410    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:29:42.260540 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260456    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc3939-4b5d-41f7-87be-0a3335c95d09-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc3939-4b5d-41f7-87be-0a3335c95d09-local-certs\") pod \"22cc3939-4b5d-41f7-87be-0a3335c95d09\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") "
Nov 02 17:29:42.260540 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260500    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-cacerts\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-cacerts\") pod \"22cc3939-4b5d-41f7-87be-0a3335c95d09\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") "
Nov 02 17:29:42.260730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260557    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-ca-configmap\") pod \"22cc3939-4b5d-41f7-87be-0a3335c95d09\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") "
Nov 02 17:29:42.260730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260595    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-kubeconfig\") pod \"22cc3939-4b5d-41f7-87be-0a3335c95d09\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") "
Nov 02 17:29:42.260730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260690    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-token\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-token\") pod \"22cc3939-4b5d-41f7-87be-0a3335c95d09\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") "
Nov 02 17:29:42.260730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260726    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-dns-cert\") pod \"22cc3939-4b5d-41f7-87be-0a3335c95d09\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") "
Nov 02 17:29:42.260919 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260771    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-kube-api-access-9nn4t\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-kube-api-access-9nn4t\") pod \"22cc3939-4b5d-41f7-87be-0a3335c95d09\" (UID: \"22cc3939-4b5d-41f7-87be-0a3335c95d09\") "
Nov 02 17:29:42.260998 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.260967    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-ca-configmap" pod "22cc3939-4b5d-41f7-87be-0a3335c95d09" (UID: "22cc3939-4b5d-41f7-87be-0a3335c95d09"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:29:42.263195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.263155    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-token" pod "22cc3939-4b5d-41f7-87be-0a3335c95d09" (UID: "22cc3939-4b5d-41f7-87be-0a3335c95d09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:29:42.263343 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.263270    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-dns-cert" pod "22cc3939-4b5d-41f7-87be-0a3335c95d09" (UID: "22cc3939-4b5d-41f7-87be-0a3335c95d09"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:29:42.263406 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.263369    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc3939-4b5d-41f7-87be-0a3335c95d09-local-certs" pod "22cc3939-4b5d-41f7-87be-0a3335c95d09" (UID: "22cc3939-4b5d-41f7-87be-0a3335c95d09"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:29:42.263480 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.263454    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-kubeconfig" pod "22cc3939-4b5d-41f7-87be-0a3335c95d09" (UID: "22cc3939-4b5d-41f7-87be-0a3335c95d09"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:29:42.263535 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.263519    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-cacerts" pod "22cc3939-4b5d-41f7-87be-0a3335c95d09" (UID: "22cc3939-4b5d-41f7-87be-0a3335c95d09"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:29:42.263587 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.263571    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-kube-api-access-9nn4t" pod "22cc3939-4b5d-41f7-87be-0a3335c95d09" (UID: "22cc3939-4b5d-41f7-87be-0a3335c95d09"). InnerVolumeSpecName "kube-api-access-9nn4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:29:42.361798 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.361767    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:29:42.361798 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.361792    4956 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-dns-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:29:42.361798 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.361803    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nn4t\" (UniqueName: \"kubernetes.io/projected/22cc3939-4b5d-41f7-87be-0a3335c95d09-kube-api-access-9nn4t\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:29:42.362221 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.361813    4956 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc3939-4b5d-41f7-87be-0a3335c95d09-local-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:29:42.362221 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.361821    4956 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-cacerts\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:29:42.362221 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.361830    4956 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-csr-ca-configmap\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:29:42.362221 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.361840    4956 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22cc3939-4b5d-41f7-87be-0a3335c95d09-istio-kubeconfig\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:29:42.546974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.546884    4956 generic.go:356] "Generic (PLEG): container finished" podID="22cc3939-4b5d-41f7-87be-0a3335c95d09" containerID="40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed" exitCode=0
Nov 02 17:29:42.546974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.546935    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"
Nov 02 17:29:42.546974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.546959    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx" event={"ID":"22cc3939-4b5d-41f7-87be-0a3335c95d09","Type":"ContainerDied","Data":"40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed"}
Nov 02 17:29:42.547293 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.547001    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx" event={"ID":"22cc3939-4b5d-41f7-87be-0a3335c95d09","Type":"ContainerDied","Data":"41f1c38ba946154387bfa7596f74b0fa72af77d41ec521a50ff676f7f5b0a9dd"}
Nov 02 17:29:42.547293 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.547020    4956 scope.go:122] "RemoveContainer" containerID="40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed"
Nov 02 17:29:42.556358 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.556339    4956 scope.go:122] "RemoveContainer" containerID="40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed"
Nov 02 17:29:42.556657 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:29:42.556638    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed\": container with ID starting with 40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed not found: ID does not exist" containerID="40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed"
Nov 02 17:29:42.556709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.556666    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed"} err="failed to get container status \"40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed\": rpc error: code = NotFound desc = could not find container \"40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed\": container with ID starting with 40576daf7af7eee19a21085c5036d7c55cab49fc93113775f3fe0caff1a35aed not found: ID does not exist"
Nov 02 17:29:42.717789 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.717752    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"]
Nov 02 17:29:42.766771 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:42.766731    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-txsvx"]
Nov 02 17:29:43.862527 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:43.862496    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cc3939-4b5d-41f7-87be-0a3335c95d09" path="/var/lib/kubelet/pods/22cc3939-4b5d-41f7-87be-0a3335c95d09/volumes"
Nov 02 17:29:51.593847 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.593814    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"]
Nov 02 17:29:51.594364 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.594322    4956 cpu_manager.go:414] "containerMap: removing container" podUID="22cc3939-4b5d-41f7-87be-0a3335c95d09" containerName="discovery"
Nov 02 17:29:51.594364 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.594339    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cc3939-4b5d-41f7-87be-0a3335c95d09" containerName="discovery"
Nov 02 17:29:51.594500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.594448    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cc3939-4b5d-41f7-87be-0a3335c95d09" containerName="discovery"
Nov 02 17:29:51.598796 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.598776    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.651614 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.651583    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:29:51.700716 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.700684    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:29:51.700916 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.700748    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 17:29:51.700916 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.700824    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-94n2w\""
Nov 02 17:29:51.701012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.700997    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 17:29:51.702350 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.702332    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"]
Nov 02 17:29:51.737865 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.737829    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgj57\" (UniqueName: \"kubernetes.io/projected/a750973c-0f78-4f19-96c1-2806eb1354da-kube-api-access-jgj57\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.737865 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.737866    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-maas-api-tls\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.738156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.737900    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-metrics-certs\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.738156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.737964    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a750973c-0f78-4f19-96c1-2806eb1354da-openshift-service-ca\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.839152 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.839086    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgj57\" (UniqueName: \"kubernetes.io/projected/a750973c-0f78-4f19-96c1-2806eb1354da-kube-api-access-jgj57\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.839152 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.839146    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-maas-api-tls\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.839406 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.839179    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-metrics-certs\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.839406 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.839201    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a750973c-0f78-4f19-96c1-2806eb1354da-openshift-service-ca\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.839836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.839806    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a750973c-0f78-4f19-96c1-2806eb1354da-openshift-service-ca\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.841918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.841898    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-maas-api-tls\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.841984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.841929    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-metrics-certs\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.896147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.896053    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgj57\" (UniqueName: \"kubernetes.io/projected/a750973c-0f78-4f19-96c1-2806eb1354da-kube-api-access-jgj57\") pod \"maas-api-67c55cc85f-7b96n\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") " pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:51.915933 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:51.915898    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:52.304244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:52.304214    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"]
Nov 02 17:29:52.306648 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:29:52.306603    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda750973c_0f78_4f19_96c1_2806eb1354da.slice/crio-afad2e74087bdfcea2031f8644cdb2a75ea64d78bc3d102813363ca6eb305d7d WatchSource:0}: Error finding container afad2e74087bdfcea2031f8644cdb2a75ea64d78bc3d102813363ca6eb305d7d: Status 404 returned error can't find the container with id afad2e74087bdfcea2031f8644cdb2a75ea64d78bc3d102813363ca6eb305d7d
Nov 02 17:29:52.593266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:52.593179    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n" event={"ID":"a750973c-0f78-4f19-96c1-2806eb1354da","Type":"ContainerStarted","Data":"afad2e74087bdfcea2031f8644cdb2a75ea64d78bc3d102813363ca6eb305d7d"}
Nov 02 17:29:56.612684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:56.612646    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n" event={"ID":"a750973c-0f78-4f19-96c1-2806eb1354da","Type":"ContainerStarted","Data":"2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a"}
Nov 02 17:29:56.613166 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:56.612720    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:29:56.730213 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:29:56.730164    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n" podStartSLOduration=2.298897079 podStartE2EDuration="5.730148594s" totalImagesPullingTime="3.431251515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:56.728802891 +0000 UTC m=+426.985833512" watchObservedRunningTime="2026-11-02 17:29:56.730148594 +0000 UTC m=+426.987179208"
Nov 02 17:29:59.062471 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:29:59.062435    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:30:02.623267 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:02.623239    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:30:03.605555 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.605513    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"]
Nov 02 17:30:03.609969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.609944    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.712693 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.712649    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"]
Nov 02 17:30:03.746277 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.746232    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-maas-api-tls\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.746484 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.746353    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861c6a94-0760-4f29-92b7-6d1a23878399-openshift-service-ca\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.746484 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.746398    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtmb\" (UniqueName: \"kubernetes.io/projected/861c6a94-0760-4f29-92b7-6d1a23878399-kube-api-access-nvtmb\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.746580 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.746513    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-metrics-certs\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.847039 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.846999    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-metrics-certs\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.847278 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.847072    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-maas-api-tls\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.847278 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.847149    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861c6a94-0760-4f29-92b7-6d1a23878399-openshift-service-ca\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.847278 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.847178    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtmb\" (UniqueName: \"kubernetes.io/projected/861c6a94-0760-4f29-92b7-6d1a23878399-kube-api-access-nvtmb\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.847757 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.847737    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861c6a94-0760-4f29-92b7-6d1a23878399-openshift-service-ca\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.849633 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.849613    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-metrics-certs\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.849730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.849653    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-maas-api-tls\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.905446 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.905364    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtmb\" (UniqueName: \"kubernetes.io/projected/861c6a94-0760-4f29-92b7-6d1a23878399-kube-api-access-nvtmb\") pod \"maas-api-5cb67577d6-p6qdn\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:03.922354 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:03.922313    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:04.296151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:04.295932    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"]
Nov 02 17:30:04.298951 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:30:04.298923    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861c6a94_0760_4f29_92b7_6d1a23878399.slice/crio-aaa253578a53a682d48199e4ea45cc9b3fe47e5be328763c350c0d3b5ab95b95 WatchSource:0}: Error finding container aaa253578a53a682d48199e4ea45cc9b3fe47e5be328763c350c0d3b5ab95b95: Status 404 returned error can't find the container with id aaa253578a53a682d48199e4ea45cc9b3fe47e5be328763c350c0d3b5ab95b95
Nov 02 17:30:04.645665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:04.645576    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn" event={"ID":"861c6a94-0760-4f29-92b7-6d1a23878399","Type":"ContainerStarted","Data":"aaa253578a53a682d48199e4ea45cc9b3fe47e5be328763c350c0d3b5ab95b95"}
Nov 02 17:30:05.652507 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:05.652463    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn" event={"ID":"861c6a94-0760-4f29-92b7-6d1a23878399","Type":"ContainerStarted","Data":"5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3"}
Nov 02 17:30:05.652983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:05.652543    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:05.771552 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:05.771498    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn" podStartSLOduration=1.937506774 podStartE2EDuration="2.771484412s" totalImagesPullingTime="833.977638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:05.768679856 +0000 UTC m=+436.025710476" watchObservedRunningTime="2026-11-02 17:30:05.771484412 +0000 UTC m=+436.028515093"
Nov 02 17:30:11.662638 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:11.662601    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 17:30:11.852676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:11.852631    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"]
Nov 02 17:30:11.852889 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:11.852864    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n" podUID="a750973c-0f78-4f19-96c1-2806eb1354da" containerName="maas-api" containerID="cri-o://2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a" gracePeriod=30
Nov 02 17:30:12.099339 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.099311    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:30:12.222011 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.221928    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a750973c-0f78-4f19-96c1-2806eb1354da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a750973c-0f78-4f19-96c1-2806eb1354da-openshift-service-ca\") pod \"a750973c-0f78-4f19-96c1-2806eb1354da\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") "
Nov 02 17:30:12.222011 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.221964    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-metrics-certs\") pod \"a750973c-0f78-4f19-96c1-2806eb1354da\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") "
Nov 02 17:30:12.222290 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.222017    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-maas-api-tls\") pod \"a750973c-0f78-4f19-96c1-2806eb1354da\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") "
Nov 02 17:30:12.222290 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.222082    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a750973c-0f78-4f19-96c1-2806eb1354da-kube-api-access-jgj57\" (UniqueName: \"kubernetes.io/projected/a750973c-0f78-4f19-96c1-2806eb1354da-kube-api-access-jgj57\") pod \"a750973c-0f78-4f19-96c1-2806eb1354da\" (UID: \"a750973c-0f78-4f19-96c1-2806eb1354da\") "
Nov 02 17:30:12.222452 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.222423    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a750973c-0f78-4f19-96c1-2806eb1354da-openshift-service-ca" pod "a750973c-0f78-4f19-96c1-2806eb1354da" (UID: "a750973c-0f78-4f19-96c1-2806eb1354da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:30:12.224595 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.224568    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a750973c-0f78-4f19-96c1-2806eb1354da-kube-api-access-jgj57" pod "a750973c-0f78-4f19-96c1-2806eb1354da" (UID: "a750973c-0f78-4f19-96c1-2806eb1354da"). InnerVolumeSpecName "kube-api-access-jgj57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:30:12.224805 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.224776    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-maas-api-tls" pod "a750973c-0f78-4f19-96c1-2806eb1354da" (UID: "a750973c-0f78-4f19-96c1-2806eb1354da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:30:12.224908 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.224835    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-metrics-certs" pod "a750973c-0f78-4f19-96c1-2806eb1354da" (UID: "a750973c-0f78-4f19-96c1-2806eb1354da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:30:12.323109 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.323071    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgj57\" (UniqueName: \"kubernetes.io/projected/a750973c-0f78-4f19-96c1-2806eb1354da-kube-api-access-jgj57\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:12.323109 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.323102    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a750973c-0f78-4f19-96c1-2806eb1354da-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:12.323109 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.323114    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:12.323367 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.323153    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a750973c-0f78-4f19-96c1-2806eb1354da-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:12.687097 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.687009    4956 generic.go:356] "Generic (PLEG): container finished" podID="a750973c-0f78-4f19-96c1-2806eb1354da" containerID="2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a" exitCode=0
Nov 02 17:30:12.687097 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.687068    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"
Nov 02 17:30:12.687097 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.687088    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n" event={"ID":"a750973c-0f78-4f19-96c1-2806eb1354da","Type":"ContainerDied","Data":"2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a"}
Nov 02 17:30:12.687643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.687156    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n" event={"ID":"a750973c-0f78-4f19-96c1-2806eb1354da","Type":"ContainerDied","Data":"afad2e74087bdfcea2031f8644cdb2a75ea64d78bc3d102813363ca6eb305d7d"}
Nov 02 17:30:12.687643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.687180    4956 scope.go:122] "RemoveContainer" containerID="2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a"
Nov 02 17:30:12.695880 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.695862    4956 scope.go:122] "RemoveContainer" containerID="2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a"
Nov 02 17:30:12.696176 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:30:12.696104    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a\": container with ID starting with 2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a not found: ID does not exist" containerID="2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a"
Nov 02 17:30:12.696176 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.696156    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a"} err="failed to get container status \"2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a\": rpc error: code = NotFound desc = could not find container \"2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a\": container with ID starting with 2bbe3c92579195bc91283e01843bed4052af7b21ad85a74ee16ba45d825a427a not found: ID does not exist"
Nov 02 17:30:12.872106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.872070    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"]
Nov 02 17:30:12.920909 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:12.920876    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c55cc85f-7b96n"]
Nov 02 17:30:13.862873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:30:13.862828    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a750973c-0f78-4f19-96c1-2806eb1354da" path="/var/lib/kubelet/pods/a750973c-0f78-4f19-96c1-2806eb1354da/volumes"
Nov 02 17:30:29.052082 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:30:29.052047    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:30:59.051849 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:30:59.051811    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:31:23.884410 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.884377    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-74hjx"]
Nov 02 17:31:23.884904 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.884790    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a750973c-0f78-4f19-96c1-2806eb1354da" containerName="maas-api"
Nov 02 17:31:23.884904 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.884808    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a750973c-0f78-4f19-96c1-2806eb1354da" containerName="maas-api"
Nov 02 17:31:23.884984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.884911    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a750973c-0f78-4f19-96c1-2806eb1354da" containerName="maas-api"
Nov 02 17:31:23.886783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.886768    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:23.931421 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.931390    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-74hjx\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:23.931585 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.931434    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnj2\" (UniqueName: \"kubernetes.io/projected/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-kube-api-access-mvnj2\") pod \"ai-gateway-controller-5d8dddb676-74hjx\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:23.988219 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.988189    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 17:31:23.988219 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.988189    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 17:31:23.988435 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.988240    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-zk7cp\""
Nov 02 17:31:23.989414 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:23.989396    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-74hjx"]
Nov 02 17:31:24.031887 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:24.031853    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-74hjx\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:24.032088 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:24.031906    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnj2\" (UniqueName: \"kubernetes.io/projected/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-kube-api-access-mvnj2\") pod \"ai-gateway-controller-5d8dddb676-74hjx\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:24.034483 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:24.034455    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-74hjx\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:24.088795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:24.088761    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnj2\" (UniqueName: \"kubernetes.io/projected/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-kube-api-access-mvnj2\") pod \"ai-gateway-controller-5d8dddb676-74hjx\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:24.198053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:24.197958    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:24.566559 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:24.566521    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-74hjx"]
Nov 02 17:31:24.569571 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:31:24.569542    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4da46f_8af3_443f_8fb4_8ca5c6372a2c.slice/crio-c172c9f34b3e9875738d5941a8cdccad10319bdb59467fc4ab6064a90c4a5c42 WatchSource:0}: Error finding container c172c9f34b3e9875738d5941a8cdccad10319bdb59467fc4ab6064a90c4a5c42: Status 404 returned error can't find the container with id c172c9f34b3e9875738d5941a8cdccad10319bdb59467fc4ab6064a90c4a5c42
Nov 02 17:31:24.977753 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:24.977715    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx" event={"ID":"da4da46f-8af3-443f-8fb4-8ca5c6372a2c","Type":"ContainerStarted","Data":"c172c9f34b3e9875738d5941a8cdccad10319bdb59467fc4ab6064a90c4a5c42"}
Nov 02 17:31:26.987684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:26.987647    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx" event={"ID":"da4da46f-8af3-443f-8fb4-8ca5c6372a2c","Type":"ContainerStarted","Data":"7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9"}
Nov 02 17:31:26.988075 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:26.987740    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:27.102442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:27.102385    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx" podStartSLOduration=2.067474801 podStartE2EDuration="4.102369294s" totalImagesPullingTime="2.034894493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:27.101520798 +0000 UTC m=+517.358551419" watchObservedRunningTime="2026-11-02 17:31:27.102369294 +0000 UTC m=+517.359399914"
Nov 02 17:31:29.052219 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:31:29.052179    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:31:30.942643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:30.942607    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"]
Nov 02 17:31:30.944697 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:30.944681    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:30.978510 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:30.978473    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b494d4-b1e9-4c28-941a-0eaee5637be8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rs7wm\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:30.978510 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:30.978509    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srvjb\" (UniqueName: \"kubernetes.io/projected/85b494d4-b1e9-4c28-941a-0eaee5637be8-kube-api-access-srvjb\") pod \"payload-pre-processing-7b98bb678c-rs7wm\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:30.996685 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:30.996654    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kx74g\""
Nov 02 17:31:31.046179 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.046141    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 17:31:31.047499 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.047475    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"]
Nov 02 17:31:31.079630 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.079593    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b494d4-b1e9-4c28-941a-0eaee5637be8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rs7wm\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:31.079630 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.079633    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srvjb\" (UniqueName: \"kubernetes.io/projected/85b494d4-b1e9-4c28-941a-0eaee5637be8-kube-api-access-srvjb\") pod \"payload-pre-processing-7b98bb678c-rs7wm\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:31.080465 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.080447    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b494d4-b1e9-4c28-941a-0eaee5637be8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rs7wm\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:31.097133 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.097087    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"]
Nov 02 17:31:31.099462 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.099444    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.137268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.137230    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srvjb\" (UniqueName: \"kubernetes.io/projected/85b494d4-b1e9-4c28-941a-0eaee5637be8-kube-api-access-srvjb\") pod \"payload-pre-processing-7b98bb678c-rs7wm\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:31.156131 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.156083    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"]
Nov 02 17:31:31.180333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.180298    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gtcjl\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.180491 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.180346    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw88f\" (UniqueName: \"kubernetes.io/projected/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-kube-api-access-mw88f\") pod \"payload-processing-55fbbbf9dc-gtcjl\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.255737 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.255701    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:31.280918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.280877    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gtcjl\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.281075 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.280917    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw88f\" (UniqueName: \"kubernetes.io/projected/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-kube-api-access-mw88f\") pod \"payload-processing-55fbbbf9dc-gtcjl\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.281845 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.281817    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gtcjl\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.337857 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.337830    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw88f\" (UniqueName: \"kubernetes.io/projected/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-kube-api-access-mw88f\") pod \"payload-processing-55fbbbf9dc-gtcjl\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.411754 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.411724    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:31.622684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.622658    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"]
Nov 02 17:31:31.625066 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:31:31.625044    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b494d4_b1e9_4c28_941a_0eaee5637be8.slice/crio-12672a586f19da99a194cda6f7d847f85caaaa4b2ace8dd5071cb929c956a264 WatchSource:0}: Error finding container 12672a586f19da99a194cda6f7d847f85caaaa4b2ace8dd5071cb929c956a264: Status 404 returned error can't find the container with id 12672a586f19da99a194cda6f7d847f85caaaa4b2ace8dd5071cb929c956a264
Nov 02 17:31:31.791642 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:31.791612    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"]
Nov 02 17:31:31.795009 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:31:31.794964    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8750e627_3fc4_4ed2_9d3f_e5a4ff8f1c44.slice/crio-93ec6a7a223fa88e89224f8750009b290e4eccecf4199a2959ced232157f3472 WatchSource:0}: Error finding container 93ec6a7a223fa88e89224f8750009b290e4eccecf4199a2959ced232157f3472: Status 404 returned error can't find the container with id 93ec6a7a223fa88e89224f8750009b290e4eccecf4199a2959ced232157f3472
Nov 02 17:31:32.008361 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:32.008317    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl" event={"ID":"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44","Type":"ContainerStarted","Data":"93ec6a7a223fa88e89224f8750009b290e4eccecf4199a2959ced232157f3472"}
Nov 02 17:31:32.009494 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:32.009473    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm" event={"ID":"85b494d4-b1e9-4c28-941a-0eaee5637be8","Type":"ContainerStarted","Data":"12672a586f19da99a194cda6f7d847f85caaaa4b2ace8dd5071cb929c956a264"}
Nov 02 17:31:35.024673 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:35.024638    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl" event={"ID":"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44","Type":"ContainerStarted","Data":"39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9"}
Nov 02 17:31:35.025173 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:35.024760    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:35.026343 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:35.026312    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm" event={"ID":"85b494d4-b1e9-4c28-941a-0eaee5637be8","Type":"ContainerStarted","Data":"3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78"}
Nov 02 17:31:35.026419 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:35.026404    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:35.138289 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:35.138207    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl" podStartSLOduration=1.500181654 podStartE2EDuration="4.138191178s" totalImagesPullingTime="2.638009524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:35.137240815 +0000 UTC m=+525.394271467" watchObservedRunningTime="2026-11-02 17:31:35.138191178 +0000 UTC m=+525.395221862"
Nov 02 17:31:35.249793 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:35.249734    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm" podStartSLOduration=2.376065701 podStartE2EDuration="5.249717413s" totalImagesPullingTime="2.873651712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:35.249157149 +0000 UTC m=+525.506187769" watchObservedRunningTime="2026-11-02 17:31:35.249717413 +0000 UTC m=+525.506748032"
Nov 02 17:31:37.993265 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:37.993235    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:38.351584 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.351505    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"]
Nov 02 17:31:38.353764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.353748    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:38.411106 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.411068    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"]
Nov 02 17:31:38.446879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.446847    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brpg\" (UniqueName: \"kubernetes.io/projected/89a58678-30c1-4403-81bd-9818ec081d07-kube-api-access-2brpg\") pod \"ai-gateway-controller-5c9b9f7776-2hch9\" (UID: \"89a58678-30c1-4403-81bd-9818ec081d07\") " pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:38.447054 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.446902    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a58678-30c1-4403-81bd-9818ec081d07-metrics-certs\") pod \"ai-gateway-controller-5c9b9f7776-2hch9\" (UID: \"89a58678-30c1-4403-81bd-9818ec081d07\") " pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:38.547883 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.547848    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brpg\" (UniqueName: \"kubernetes.io/projected/89a58678-30c1-4403-81bd-9818ec081d07-kube-api-access-2brpg\") pod \"ai-gateway-controller-5c9b9f7776-2hch9\" (UID: \"89a58678-30c1-4403-81bd-9818ec081d07\") " pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:38.548059 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.547918    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a58678-30c1-4403-81bd-9818ec081d07-metrics-certs\") pod \"ai-gateway-controller-5c9b9f7776-2hch9\" (UID: \"89a58678-30c1-4403-81bd-9818ec081d07\") " pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:38.550500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.550480    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a58678-30c1-4403-81bd-9818ec081d07-metrics-certs\") pod \"ai-gateway-controller-5c9b9f7776-2hch9\" (UID: \"89a58678-30c1-4403-81bd-9818ec081d07\") " pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:38.604966 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.604890    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brpg\" (UniqueName: \"kubernetes.io/projected/89a58678-30c1-4403-81bd-9818ec081d07-kube-api-access-2brpg\") pod \"ai-gateway-controller-5c9b9f7776-2hch9\" (UID: \"89a58678-30c1-4403-81bd-9818ec081d07\") " pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:38.665183 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:38.665144    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:39.034745 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:39.034715    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"]
Nov 02 17:31:39.037402 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:31:39.037370    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a58678_30c1_4403_81bd_9818ec081d07.slice/crio-46a6a8fa6f9bbd21d34d702f92437630ced4a746a02121d3d7e98c4229421c9c WatchSource:0}: Error finding container 46a6a8fa6f9bbd21d34d702f92437630ced4a746a02121d3d7e98c4229421c9c: Status 404 returned error can't find the container with id 46a6a8fa6f9bbd21d34d702f92437630ced4a746a02121d3d7e98c4229421c9c
Nov 02 17:31:39.043446 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:39.043420    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9" event={"ID":"89a58678-30c1-4403-81bd-9818ec081d07","Type":"ContainerStarted","Data":"46a6a8fa6f9bbd21d34d702f92437630ced4a746a02121d3d7e98c4229421c9c"}
Nov 02 17:31:40.048984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:40.048951    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9" event={"ID":"89a58678-30c1-4403-81bd-9818ec081d07","Type":"ContainerStarted","Data":"f1945e52a0efc3dab7a709c2e90c64b3a7bcfd2c12653a2dac28b1870b6e67cf"}
Nov 02 17:31:40.049409 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:40.049088    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:40.163141 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:40.163006    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9" podStartSLOduration=1.369545737 podStartE2EDuration="2.16299159s" totalImagesPullingTime="793.445853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:40.161841922 +0000 UTC m=+530.418872542" watchObservedRunningTime="2026-11-02 17:31:40.16299159 +0000 UTC m=+530.420022201"
Nov 02 17:31:46.034172 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:46.034141    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 17:31:46.034575 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:46.034304    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 17:31:51.056394 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.056362    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5c9b9f7776-2hch9"
Nov 02 17:31:51.221579 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.221541    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-74hjx"]
Nov 02 17:31:51.221982 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.221939    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx" podUID="da4da46f-8af3-443f-8fb4-8ca5c6372a2c" containerName="manager" containerID="cri-o://7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9" gracePeriod=30
Nov 02 17:31:51.472158 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.472132    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:51.556052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.556015    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-kube-api-access-mvnj2\" (UniqueName: \"kubernetes.io/projected/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-kube-api-access-mvnj2\") pod \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") "
Nov 02 17:31:51.556272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.556077    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-metrics-certs\") pod \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\" (UID: \"da4da46f-8af3-443f-8fb4-8ca5c6372a2c\") "
Nov 02 17:31:51.558551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.558518    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-kube-api-access-mvnj2" pod "da4da46f-8af3-443f-8fb4-8ca5c6372a2c" (UID: "da4da46f-8af3-443f-8fb4-8ca5c6372a2c"). InnerVolumeSpecName "kube-api-access-mvnj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:31:51.558635 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.558518    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-metrics-certs" pod "da4da46f-8af3-443f-8fb4-8ca5c6372a2c" (UID: "da4da46f-8af3-443f-8fb4-8ca5c6372a2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:51.657198 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.657074    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:31:51.657198 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:51.657115    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvnj2\" (UniqueName: \"kubernetes.io/projected/da4da46f-8af3-443f-8fb4-8ca5c6372a2c-kube-api-access-mvnj2\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:31:52.099299 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.099257    4956 generic.go:356] "Generic (PLEG): container finished" podID="da4da46f-8af3-443f-8fb4-8ca5c6372a2c" containerID="7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9" exitCode=0
Nov 02 17:31:52.099794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.099328    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx"
Nov 02 17:31:52.099794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.099337    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx" event={"ID":"da4da46f-8af3-443f-8fb4-8ca5c6372a2c","Type":"ContainerDied","Data":"7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9"}
Nov 02 17:31:52.099794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.099417    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-74hjx" event={"ID":"da4da46f-8af3-443f-8fb4-8ca5c6372a2c","Type":"ContainerDied","Data":"c172c9f34b3e9875738d5941a8cdccad10319bdb59467fc4ab6064a90c4a5c42"}
Nov 02 17:31:52.099794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.099439    4956 scope.go:122] "RemoveContainer" containerID="7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9"
Nov 02 17:31:52.109370 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.109349    4956 scope.go:122] "RemoveContainer" containerID="7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9"
Nov 02 17:31:52.109738 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:31:52.109716    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9\": container with ID starting with 7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9 not found: ID does not exist" containerID="7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9"
Nov 02 17:31:52.109775 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.109750    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9"} err="failed to get container status \"7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9\": rpc error: code = NotFound desc = could not find container \"7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9\": container with ID starting with 7c078af92d01662792dc5f01190ea545daebdc7c0db86884e6ffe45a780d7dd9 not found: ID does not exist"
Nov 02 17:31:52.261690 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.261654    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-74hjx"]
Nov 02 17:31:52.311303 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:52.311267    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-74hjx"]
Nov 02 17:31:53.863293 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:31:53.863259    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4da46f-8af3-443f-8fb4-8ca5c6372a2c" path="/var/lib/kubelet/pods/da4da46f-8af3-443f-8fb4-8ca5c6372a2c/volumes"
Nov 02 17:31:59.051469 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:31:59.051428    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:32:00.524359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.524317    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"]
Nov 02 17:32:00.524769 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.524668    4956 cpu_manager.go:414] "containerMap: removing container" podUID="da4da46f-8af3-443f-8fb4-8ca5c6372a2c" containerName="manager"
Nov 02 17:32:00.524769 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.524679    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4da46f-8af3-443f-8fb4-8ca5c6372a2c" containerName="manager"
Nov 02 17:32:00.524769 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.524749    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4da46f-8af3-443f-8fb4-8ca5c6372a2c" containerName="manager"
Nov 02 17:32:00.531812 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.531788    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.586708 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.586678    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"]
Nov 02 17:32:00.599183 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.599150    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 17:32:00.633859 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.633826    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 17:32:00.634066 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.633993    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5r2ch\""
Nov 02 17:32:00.634066 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.633993    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 17:32:00.636561 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.636531    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.636625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.636610    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.636678 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.636642    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.636768 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.636745    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917a756a-a002-4e8c-ae59-65257b63ef58-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.636839 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.636799    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5sx\" (UniqueName: \"kubernetes.io/projected/917a756a-a002-4e8c-ae59-65257b63ef58-kube-api-access-cv5sx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.636894 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.636864    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.737879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.737839    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.737879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.737879    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917a756a-a002-4e8c-ae59-65257b63ef58-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.738171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.737904    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5sx\" (UniqueName: \"kubernetes.io/projected/917a756a-a002-4e8c-ae59-65257b63ef58-kube-api-access-cv5sx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.738171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.737936    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.738171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.738145    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.738347 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.738220    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.738429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.738412    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.738481 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.738411    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.738564 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.738543    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.740438 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.740416    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917a756a-a002-4e8c-ae59-65257b63ef58-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.740820 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.740798    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917a756a-a002-4e8c-ae59-65257b63ef58-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.795322 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.795255    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5sx\" (UniqueName: \"kubernetes.io/projected/917a756a-a002-4e8c-ae59-65257b63ef58-kube-api-access-cv5sx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25\" (UID: \"917a756a-a002-4e8c-ae59-65257b63ef58\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:00.847500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:00.847458    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:01.230615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:01.230589    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"]
Nov 02 17:32:01.233257 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:32:01.233230    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917a756a_a002_4e8c_ae59_65257b63ef58.slice/crio-928734ce7b54f39de6fc504ed4404337183aedd08f644925fe042e76a2450793 WatchSource:0}: Error finding container 928734ce7b54f39de6fc504ed4404337183aedd08f644925fe042e76a2450793: Status 404 returned error can't find the container with id 928734ce7b54f39de6fc504ed4404337183aedd08f644925fe042e76a2450793
Nov 02 17:32:02.144445 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:02.144397    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25" event={"ID":"917a756a-a002-4e8c-ae59-65257b63ef58","Type":"ContainerStarted","Data":"928734ce7b54f39de6fc504ed4404337183aedd08f644925fe042e76a2450793"}
Nov 02 17:32:04.157406 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:04.157368    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25" event={"ID":"917a756a-a002-4e8c-ae59-65257b63ef58","Type":"ContainerStarted","Data":"f0bc2a8f180fbf119b01daa5e791322319f43fc892afe90202d87ed13a960891"}
Nov 02 17:32:04.272577 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:04.272522    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25" podStartSLOduration=1.959712567 podStartE2EDuration="4.272508274s" totalImagesPullingTime="2.312795707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:04.270942094 +0000 UTC m=+554.527972718" watchObservedRunningTime="2026-11-02 17:32:04.272508274 +0000 UTC m=+554.529538893"
Nov 02 17:32:04.848053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:04.848013    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:07.722775 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.722735    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"]
Nov 02 17:32:07.726564 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.726541    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.778905 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.778865    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 17:32:07.803761 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.803705    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.803950 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.803789    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aadf12b-de32-492e-ac42-5d9f61da5175-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.803950 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.803869    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.803950 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.803894    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.803950 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.803922    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.804100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.804040    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vklmj\" (UniqueName: \"kubernetes.io/projected/4aadf12b-de32-492e-ac42-5d9f61da5175-kube-api-access-vklmj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.829297 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.829265    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"]
Nov 02 17:32:07.905046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905008    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vklmj\" (UniqueName: \"kubernetes.io/projected/4aadf12b-de32-492e-ac42-5d9f61da5175-kube-api-access-vklmj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905065    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905114    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aadf12b-de32-492e-ac42-5d9f61da5175-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905189    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905222    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905262    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905597    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905634    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.905686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.905670    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.907661 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.907641    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aadf12b-de32-492e-ac42-5d9f61da5175-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.907942 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.907926    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aadf12b-de32-492e-ac42-5d9f61da5175-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:07.962644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:07.962615    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vklmj\" (UniqueName: \"kubernetes.io/projected/4aadf12b-de32-492e-ac42-5d9f61da5175-kube-api-access-vklmj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p5q58\" (UID: \"4aadf12b-de32-492e-ac42-5d9f61da5175\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:08.039803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:08.039710    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:08.414548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:08.414464    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"]
Nov 02 17:32:08.417341 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:32:08.417316    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aadf12b_de32_492e_ac42_5d9f61da5175.slice/crio-c7a70f3c8b037e8f4b229ae7b969d65df9c985c250b8d5c0bdce965fd11ad6b9 WatchSource:0}: Error finding container c7a70f3c8b037e8f4b229ae7b969d65df9c985c250b8d5c0bdce965fd11ad6b9: Status 404 returned error can't find the container with id c7a70f3c8b037e8f4b229ae7b969d65df9c985c250b8d5c0bdce965fd11ad6b9
Nov 02 17:32:09.180080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:09.180043    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58" event={"ID":"4aadf12b-de32-492e-ac42-5d9f61da5175","Type":"ContainerStarted","Data":"3b8ec45837d50c4da0a15c2e815da4d879d482c94480fb92ffb7cdf4941ff67e"}
Nov 02 17:32:09.180469 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:09.180103    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58" event={"ID":"4aadf12b-de32-492e-ac42-5d9f61da5175","Type":"ContainerStarted","Data":"c7a70f3c8b037e8f4b229ae7b969d65df9c985c250b8d5c0bdce965fd11ad6b9"}
Nov 02 17:32:09.299080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:09.298973    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58" podStartSLOduration=1.698226029 podStartE2EDuration="2.298956833s" totalImagesPullingTime="600.730804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:09.296813239 +0000 UTC m=+559.553843858" watchObservedRunningTime="2026-11-02 17:32:09.298956833 +0000 UTC m=+559.555987453"
Nov 02 17:32:10.040285 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:10.040254    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:10.847599 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:10.847559    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:10.860656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:10.860633    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:11.200210 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.200107    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25"
Nov 02 17:32:11.276561 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.276528    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"]
Nov 02 17:32:11.279919 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.279898    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.333498 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.333466    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 17:32:11.337848 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.337822    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"]
Nov 02 17:32:11.437871 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.437841    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkdv\" (UniqueName: \"kubernetes.io/projected/c36888bb-ff8a-4945-8a50-5aa34718d80c-kube-api-access-gqkdv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.438037 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.437898    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.438037 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.437960    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.438037 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.437975    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.438171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.438094    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.438171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.438151    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36888bb-ff8a-4945-8a50-5aa34718d80c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.538793 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.538754    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.538793 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.538794    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36888bb-ff8a-4945-8a50-5aa34718d80c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.539060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.538816    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkdv\" (UniqueName: \"kubernetes.io/projected/c36888bb-ff8a-4945-8a50-5aa34718d80c-kube-api-access-gqkdv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.539060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.538859    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.539060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.539023    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.539257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.539060    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.539347 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.539327    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.539457 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.539432    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.539574 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.539436    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.541405 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.541371    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c36888bb-ff8a-4945-8a50-5aa34718d80c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.541501 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.541466    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c36888bb-ff8a-4945-8a50-5aa34718d80c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.595839 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.595805    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkdv\" (UniqueName: \"kubernetes.io/projected/c36888bb-ff8a-4945-8a50-5aa34718d80c-kube-api-access-gqkdv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b85dq\" (UID: \"c36888bb-ff8a-4945-8a50-5aa34718d80c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:11.892754 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:11.892665    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:12.262974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:12.262949    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"]
Nov 02 17:32:12.265398 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:32:12.265372    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36888bb_ff8a_4945_8a50_5aa34718d80c.slice/crio-bd6d7e6b16658953c827c0cddbd2b7c131464dedcb4e524fd81dba8ddfb38872 WatchSource:0}: Error finding container bd6d7e6b16658953c827c0cddbd2b7c131464dedcb4e524fd81dba8ddfb38872: Status 404 returned error can't find the container with id bd6d7e6b16658953c827c0cddbd2b7c131464dedcb4e524fd81dba8ddfb38872
Nov 02 17:32:13.199690 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:13.199590    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq" event={"ID":"c36888bb-ff8a-4945-8a50-5aa34718d80c","Type":"ContainerStarted","Data":"10732b6fd9d0daf5e05b3491c1b04d2d2635066db4f6fa574e483ede9a4fc13b"}
Nov 02 17:32:13.199690 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:13.199649    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq" event={"ID":"c36888bb-ff8a-4945-8a50-5aa34718d80c","Type":"ContainerStarted","Data":"bd6d7e6b16658953c827c0cddbd2b7c131464dedcb4e524fd81dba8ddfb38872"}
Nov 02 17:32:13.315153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:13.315080    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq" podStartSLOduration=1.701579269 podStartE2EDuration="2.315059184s" totalImagesPullingTime="613.479915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:13.314079406 +0000 UTC m=+563.571110025" watchObservedRunningTime="2026-11-02 17:32:13.315059184 +0000 UTC m=+563.572089806"
Nov 02 17:32:13.893287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:13.893249    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:15.016759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.016722    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"]
Nov 02 17:32:15.020245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.020227    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.121717 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.121681    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 17:32:15.123187 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.123166    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"]
Nov 02 17:32:15.172819 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.172784    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9963c070-7c0d-4440-a22a-13d28a2648d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.172819 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.172825    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln57g\" (UniqueName: \"kubernetes.io/projected/9963c070-7c0d-4440-a22a-13d28a2648d9-kube-api-access-ln57g\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.173044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.172848    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.173044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.172998    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.173044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.173037    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.173181 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.173160    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274348 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274258    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9963c070-7c0d-4440-a22a-13d28a2648d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274348 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274300    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln57g\" (UniqueName: \"kubernetes.io/projected/9963c070-7c0d-4440-a22a-13d28a2648d9-kube-api-access-ln57g\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274348 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274329    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274384    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274416    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274466    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274850    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274895    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.274953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.274906    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.276987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.276957    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9963c070-7c0d-4440-a22a-13d28a2648d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.277082 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.276993    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9963c070-7c0d-4440-a22a-13d28a2648d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.334271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.334231    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln57g\" (UniqueName: \"kubernetes.io/projected/9963c070-7c0d-4440-a22a-13d28a2648d9-kube-api-access-ln57g\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s4975\" (UID: \"9963c070-7c0d-4440-a22a-13d28a2648d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:15.633058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:15.632962    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:16.006288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:16.006131    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"]
Nov 02 17:32:16.008525 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:32:16.008494    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9963c070_7c0d_4440_a22a_13d28a2648d9.slice/crio-2b27e49bd46ab507adbde1e7ec7ca0564fda002f1071968cf20eddca48697073 WatchSource:0}: Error finding container 2b27e49bd46ab507adbde1e7ec7ca0564fda002f1071968cf20eddca48697073: Status 404 returned error can't find the container with id 2b27e49bd46ab507adbde1e7ec7ca0564fda002f1071968cf20eddca48697073
Nov 02 17:32:16.215792 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:16.215759    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975" event={"ID":"9963c070-7c0d-4440-a22a-13d28a2648d9","Type":"ContainerStarted","Data":"2b27e49bd46ab507adbde1e7ec7ca0564fda002f1071968cf20eddca48697073"}
Nov 02 17:32:17.221957 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:17.221916    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975" event={"ID":"9963c070-7c0d-4440-a22a-13d28a2648d9","Type":"ContainerStarted","Data":"b43d19df57834abee2299edbac2070c4c4a5edec30f6fc2908f92ef9c2d768c1"}
Nov 02 17:32:17.337707 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:17.337654    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975" podStartSLOduration=2.725372395 podStartE2EDuration="3.33764013s" totalImagesPullingTime="612.267735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:17.335831053 +0000 UTC m=+567.592861673" watchObservedRunningTime="2026-11-02 17:32:17.33764013 +0000 UTC m=+567.594670750"
Nov 02 17:32:17.633502 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:17.633417    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:18.040302 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.040268    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:18.052779 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.052749    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:18.237631 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.237602    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p5q58"
Nov 02 17:32:18.676938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.676898    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"]
Nov 02 17:32:18.680438 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.680422    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.732478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.732440    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 17:32:18.732478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.732447    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\""
Nov 02 17:32:18.741756 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.741726    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"]
Nov 02 17:32:18.808487 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808459    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808637 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808498    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808637 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808622    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808749 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808656    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808749 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808672    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808814 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808800    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808847 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808831    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bd9608-761b-4a88-8641-4a56fb5d092c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808882 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808852    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfxd\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-kube-api-access-khfxd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.808882 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.808873    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909387    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bd9608-761b-4a88-8641-4a56fb5d092c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909420    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfxd\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-kube-api-access-khfxd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909441    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909573    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909623    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909750 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909702    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909750 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909735    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909847 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909760    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909847 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909814    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.909979 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.909947    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.910063 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.910041    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.910247 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.910217    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bd9608-761b-4a88-8641-4a56fb5d092c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.910346 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.910226    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.910408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.910352    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.912058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.912038    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.912058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.912053    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.966249 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.966203    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.966389 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.966270    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfxd\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-kube-api-access-khfxd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-drckw\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:18.993167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:18.993105    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:19.417034 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:19.416951    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"]
Nov 02 17:32:19.420263 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:32:19.420233    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bd9608_761b_4a88_8641_4a56fb5d092c.slice/crio-ba262dcb88208d615b0af1fddd1dc3b682393faed4e18d05dd37830a25ab5de2 WatchSource:0}: Error finding container ba262dcb88208d615b0af1fddd1dc3b682393faed4e18d05dd37830a25ab5de2: Status 404 returned error can't find the container with id ba262dcb88208d615b0af1fddd1dc3b682393faed4e18d05dd37830a25ab5de2
Nov 02 17:32:20.236563 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:20.236527    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" event={"ID":"98bd9608-761b-4a88-8641-4a56fb5d092c","Type":"ContainerStarted","Data":"ba262dcb88208d615b0af1fddd1dc3b682393faed4e18d05dd37830a25ab5de2"}
Nov 02 17:32:21.893624 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:21.893590    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:21.909433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:21.909401    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:22.260192 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:22.260156    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b85dq"
Nov 02 17:32:23.261455 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:23.261421    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:32:23.261914 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:23.261501    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:32:23.261914 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:23.261548    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:32:24.260134 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:24.260081    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" event={"ID":"98bd9608-761b-4a88-8641-4a56fb5d092c","Type":"ContainerStarted","Data":"681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896"}
Nov 02 17:32:24.420869 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:24.420779    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podStartSLOduration=2.582531334 podStartE2EDuration="6.420765336s" totalImagesPullingTime="3.838234002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:24.383863238 +0000 UTC m=+574.640893861" watchObservedRunningTime="2026-11-02 17:32:24.420765336 +0000 UTC m=+574.677795955"
Nov 02 17:32:24.993795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:24.993751    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:24.995549 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:24.995520    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:32:24.995672 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:24.995578    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 17:32:25.638139 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:25.635552    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:25.661543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:25.661514    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:25.994015 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:25.993976    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:32:25.994244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:25.994041    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 17:32:26.280897 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:26.280810    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s4975"
Nov 02 17:32:26.993968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:26.993938    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:32:26.994391 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:26.993996    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 17:32:27.765480 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.765444    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"]
Nov 02 17:32:27.769249 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.769230    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.804245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.804208    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bc4\" (UniqueName: \"kubernetes.io/projected/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-kube-api-access-z7bc4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.804398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.804256    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.804398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.804279    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.804398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.804357    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.804398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.804376    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.804543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.804486    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.832147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.825898    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"]
Nov 02 17:32:27.872894 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.872863    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 17:32:27.909981 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.909941    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910210 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.909991    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.910246    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.910280    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910383 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.910331    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910444 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.910429    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bc4\" (UniqueName: \"kubernetes.io/projected/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-kube-api-access-z7bc4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910662 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.910631    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910808 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.910668    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.910870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.910700    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.912675 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.912654    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.912918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.912899    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.968139 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.968085    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bc4\" (UniqueName: \"kubernetes.io/projected/dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c-kube-api-access-z7bc4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc\" (UID: \"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:27.994447 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.994404    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:32:27.994820 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:27.994488    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 17:32:28.082741 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:28.082648    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:28.474839 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:28.474811    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"]
Nov 02 17:32:28.477178 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:32:28.477149    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9ee5aa_c5bb_4ea4_aa9d_84ba8b8a512c.slice/crio-08d9b242f6cbba9bb3fb7ad133d1947077ed9680ac8ae8cef4a90ead615211f7 WatchSource:0}: Error finding container 08d9b242f6cbba9bb3fb7ad133d1947077ed9680ac8ae8cef4a90ead615211f7: Status 404 returned error can't find the container with id 08d9b242f6cbba9bb3fb7ad133d1947077ed9680ac8ae8cef4a90ead615211f7
Nov 02 17:32:28.993558 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:28.993515    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:32:28.993715 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:28.993568    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 17:32:29.078933 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:32:29.078807    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:32:29.286550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:29.286462    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc" event={"ID":"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c","Type":"ContainerStarted","Data":"a35f834eaa19c3f75820a4765503a5726e400634b040468de9773d6aa6da3f0e"}
Nov 02 17:32:29.286550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:29.286513    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc" event={"ID":"dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c","Type":"ContainerStarted","Data":"08d9b242f6cbba9bb3fb7ad133d1947077ed9680ac8ae8cef4a90ead615211f7"}
Nov 02 17:32:29.401619 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:29.401560    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc" podStartSLOduration=1.7747577479999999 podStartE2EDuration="2.401543726s" totalImagesPullingTime="626.785978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:29.400283454 +0000 UTC m=+579.657314075" watchObservedRunningTime="2026-11-02 17:32:29.401543726 +0000 UTC m=+579.658574346"
Nov 02 17:32:29.994408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:29.994363    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:32:29.994589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:29.994451    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 17:32:30.084066 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:30.084025    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:31.458906 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.458868    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"]
Nov 02 17:32:31.463052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.463027    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.544900 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.544860    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssrs\" (UniqueName: \"kubernetes.io/projected/5e711d1c-6058-4345-9189-de8749615f2c-kube-api-access-lssrs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.545095 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.544970    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.545095 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.545040    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.545212 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.545197    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e711d1c-6058-4345-9189-de8749615f2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.545274 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.545256    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.545352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.545328    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.566010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.565973    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 17:32:31.568539 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.568504    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"]
Nov 02 17:32:31.646316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646274    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e711d1c-6058-4345-9189-de8749615f2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.646515 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646359    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.646515 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646392    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.646515 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646443    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssrs\" (UniqueName: \"kubernetes.io/projected/5e711d1c-6058-4345-9189-de8749615f2c-kube-api-access-lssrs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.646515 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646487    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.646771 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646536    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.646863 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646839    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.646952 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.646933    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.647151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.647107    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.650099 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.650072    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e711d1c-6058-4345-9189-de8749615f2c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.651224 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.651198    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e711d1c-6058-4345-9189-de8749615f2c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.704335 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.704300    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssrs\" (UniqueName: \"kubernetes.io/projected/5e711d1c-6058-4345-9189-de8749615f2c-kube-api-access-lssrs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct\" (UID: \"5e711d1c-6058-4345-9189-de8749615f2c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:31.788972 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:31.788868    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:32.034416 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:32.034368    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:32:32.034630 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:32.034455    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:32:32.186168 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:32.184348    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"]
Nov 02 17:32:32.191329 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:32:32.191271    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e711d1c_6058_4345_9189_de8749615f2c.slice/crio-a07b19d0db9577b1af79c5b545c4f5e21f0e5bc5174e53072e48f76e92317d49 WatchSource:0}: Error finding container a07b19d0db9577b1af79c5b545c4f5e21f0e5bc5174e53072e48f76e92317d49: Status 404 returned error can't find the container with id a07b19d0db9577b1af79c5b545c4f5e21f0e5bc5174e53072e48f76e92317d49
Nov 02 17:32:32.302925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:32.302888    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct" event={"ID":"5e711d1c-6058-4345-9189-de8749615f2c","Type":"ContainerStarted","Data":"a07b19d0db9577b1af79c5b545c4f5e21f0e5bc5174e53072e48f76e92317d49"}
Nov 02 17:32:32.394013 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:32.393981    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:32.394391 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:32.394368    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:32.395155 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:32.395110    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 17:32:33.309222 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:33.309178    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct" event={"ID":"5e711d1c-6058-4345-9189-de8749615f2c","Type":"ContainerStarted","Data":"bcd6106c7483203c4a77f5e3a8a8156d35a5c95cb0722f95e2039f35553dc0ca"}
Nov 02 17:32:33.424394 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:33.424339    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct" podStartSLOduration=1.817459825 podStartE2EDuration="2.424323178s" totalImagesPullingTime="606.863353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:33.423607208 +0000 UTC m=+583.680637844" watchObservedRunningTime="2026-11-02 17:32:33.424323178 +0000 UTC m=+583.681353797"
Nov 02 17:32:33.789485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:33.789448    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:38.083969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:38.083923    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:38.096816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:38.096786    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:38.344333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:38.344246    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc"
Nov 02 17:32:41.789023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:41.788989    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:41.802081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:41.802050    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:42.362948 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:42.362911    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:32:49.915531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:49.915500    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:32:49.915930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:32:49.915601    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:32:59.052629 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:32:59.052593    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:33:29.054538 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:29.054492    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:33:39.156497 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.156463    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"]
Nov 02 17:33:39.160026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.160000    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169299 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169269    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-credential-socket\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169313    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-data\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169420    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-socket\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169558 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169466    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-istio-token\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169558 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169493    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787ce83b-686e-453e-90d5-61381a488a68-istiod-ca-cert\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169666 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169611    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vml5v\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-kube-api-access-vml5v\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169679    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-certs\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169712    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-envoy\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.169821 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.169738    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787ce83b-686e-453e-90d5-61381a488a68-istio-podinfo\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.226453 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.226415    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-135f915e-openshift-default-dockercfg-24zqc\""
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.270852    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-credential-socket\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.270897    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-data\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271005    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-socket\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271053    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-istio-token\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271079    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787ce83b-686e-453e-90d5-61381a488a68-istiod-ca-cert\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271153    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vml5v\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-kube-api-access-vml5v\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271182    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-certs\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271213    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-envoy\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271259    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787ce83b-686e-453e-90d5-61381a488a68-istio-podinfo\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.271100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271332    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-credential-socket\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.272268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.271846    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-socket\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.272268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.272000    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787ce83b-686e-453e-90d5-61381a488a68-istiod-ca-cert\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.272268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.272010    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-certs\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.272430 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.272339    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-data\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.274516 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.274494    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-envoy\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.274707 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.274685    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787ce83b-686e-453e-90d5-61381a488a68-istio-podinfo\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.275809 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.275783    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"]
Nov 02 17:33:39.330628 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.330595    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-istio-token\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.330934 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.330913    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vml5v\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-kube-api-access-vml5v\") pod \"e2e-derive-135f915e-openshift-default-5957794c65-gpv4w\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") " pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.473306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.473269    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:39.895762 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.895728    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"]
Nov 02 17:33:39.898225 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:39.898196    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787ce83b_686e_453e_90d5_61381a488a68.slice/crio-70fb88ccaa9b9435cf2aa97a29c2b3466ece90605f2c04fa1b09a03b96ed0194 WatchSource:0}: Error finding container 70fb88ccaa9b9435cf2aa97a29c2b3466ece90605f2c04fa1b09a03b96ed0194: Status 404 returned error can't find the container with id 70fb88ccaa9b9435cf2aa97a29c2b3466ece90605f2c04fa1b09a03b96ed0194
Nov 02 17:33:39.900307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.900273    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:33:39.900420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.900343    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:33:39.900420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:39.900370    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:33:40.619391 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.619358    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" event={"ID":"787ce83b-686e-453e-90d5-61381a488a68","Type":"ContainerStarted","Data":"09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275"}
Nov 02 17:33:40.619820 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.619406    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" event={"ID":"787ce83b-686e-453e-90d5-61381a488a68","Type":"ContainerStarted","Data":"70fb88ccaa9b9435cf2aa97a29c2b3466ece90605f2c04fa1b09a03b96ed0194"}
Nov 02 17:33:40.702191 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.702160    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"]
Nov 02 17:33:40.705634 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.705617    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:40.735527 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.735478    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podStartSLOduration=1.735459467 podStartE2EDuration="1.735459467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:40.73458838 +0000 UTC m=+650.991618991" watchObservedRunningTime="2026-11-02 17:33:40.735459467 +0000 UTC m=+650.992490081"
Nov 02 17:33:40.760738 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.760705    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cc12ac52\""
Nov 02 17:33:40.764275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.764239    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"]
Nov 02 17:33:40.781585 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.781560    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cc12ac52-dockercfg-vp467\""
Nov 02 17:33:40.786174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.786108    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131778ce-a76b-484d-b605-c9997e59ce59-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:40.786314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.786190    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52mk\" (UniqueName: \"kubernetes.io/projected/131778ce-a76b-484d-b605-c9997e59ce59-kube-api-access-p52mk\") pod \"payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:40.859420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.859392    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"]
Nov 02 17:33:40.863431 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.863408    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:40.887524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.887489    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131778ce-a76b-484d-b605-c9997e59ce59-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:40.887524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.887526    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52mk\" (UniqueName: \"kubernetes.io/projected/131778ce-a76b-484d-b605-c9997e59ce59-kube-api-access-p52mk\") pod \"payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:40.887751 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.887592    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5t6\" (UniqueName: \"kubernetes.io/projected/a98853fd-8696-41df-b917-6634ba970658-kube-api-access-7w5t6\") pod \"payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:40.887751 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.887631    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98853fd-8696-41df-b917-6634ba970658-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:40.888302 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.888276    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131778ce-a76b-484d-b605-c9997e59ce59-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:40.919891 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.919842    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"]
Nov 02 17:33:40.944368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.944329    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52mk\" (UniqueName: \"kubernetes.io/projected/131778ce-a76b-484d-b605-c9997e59ce59-kube-api-access-p52mk\") pod \"payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:40.988598 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.988527    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t6\" (UniqueName: \"kubernetes.io/projected/a98853fd-8696-41df-b917-6634ba970658-kube-api-access-7w5t6\") pod \"payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:40.988598 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.988594    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98853fd-8696-41df-b917-6634ba970658-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:40.989336 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:40.989316    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98853fd-8696-41df-b917-6634ba970658-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:41.019350 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.019319    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:41.045361 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.045332    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5t6\" (UniqueName: \"kubernetes.io/projected/a98853fd-8696-41df-b917-6634ba970658-kube-api-access-7w5t6\") pod \"payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:41.178085 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.177995    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:41.389344 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.389320    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"]
Nov 02 17:33:41.391280 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:41.391247    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131778ce_a76b_484d_b605_c9997e59ce59.slice/crio-f3ab0eaeb4d47113805d9ae36fe5c149d031683473b2914a8ee6daf70f53afba WatchSource:0}: Error finding container f3ab0eaeb4d47113805d9ae36fe5c149d031683473b2914a8ee6daf70f53afba: Status 404 returned error can't find the container with id f3ab0eaeb4d47113805d9ae36fe5c149d031683473b2914a8ee6daf70f53afba
Nov 02 17:33:41.473655 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.473619    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:41.478666 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.478643    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:41.553830 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.553789    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"]
Nov 02 17:33:41.556067 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:41.556037    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98853fd_8696_41df_b917_6634ba970658.slice/crio-06a11785e763af03801e32fde42f1a5e69c8b31d6a93e3a175debe3f3e370737 WatchSource:0}: Error finding container 06a11785e763af03801e32fde42f1a5e69c8b31d6a93e3a175debe3f3e370737: Status 404 returned error can't find the container with id 06a11785e763af03801e32fde42f1a5e69c8b31d6a93e3a175debe3f3e370737
Nov 02 17:33:41.626730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.626694    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52" event={"ID":"a98853fd-8696-41df-b917-6634ba970658","Type":"ContainerStarted","Data":"6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437"}
Nov 02 17:33:41.627150 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.626758    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52" event={"ID":"a98853fd-8696-41df-b917-6634ba970658","Type":"ContainerStarted","Data":"06a11785e763af03801e32fde42f1a5e69c8b31d6a93e3a175debe3f3e370737"}
Nov 02 17:33:41.627150 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.626904    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:41.628420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.628396    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2" event={"ID":"131778ce-a76b-484d-b605-c9997e59ce59","Type":"ContainerStarted","Data":"a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38"}
Nov 02 17:33:41.628571 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.628440    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2" event={"ID":"131778ce-a76b-484d-b605-c9997e59ce59","Type":"ContainerStarted","Data":"f3ab0eaeb4d47113805d9ae36fe5c149d031683473b2914a8ee6daf70f53afba"}
Nov 02 17:33:41.628571 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.628489    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:41.628571 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.628548    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:41.629534 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.629515    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:33:41.747819 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.747765    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52" podStartSLOduration=1.747749764 podStartE2EDuration="1.747749764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:41.745856494 +0000 UTC m=+652.002887115" watchObservedRunningTime="2026-11-02 17:33:41.747749764 +0000 UTC m=+652.004780383"
Nov 02 17:33:41.861679 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:41.861621    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2" podStartSLOduration=1.861607569 podStartE2EDuration="1.861607569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:41.859322801 +0000 UTC m=+652.116353422" watchObservedRunningTime="2026-11-02 17:33:41.861607569 +0000 UTC m=+652.118638189"
Nov 02 17:33:42.322081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.322046    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"]
Nov 02 17:33:42.325769 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.325744    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.377726 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.377699    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-aa5b913c-serving-cert\""
Nov 02 17:33:42.389335 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.389305    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"]
Nov 02 17:33:42.401447 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.401421    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-metrics-certs\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.401595 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.401480    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxcx\" (UniqueName: \"kubernetes.io/projected/044b34c2-4296-430e-82a0-c98fbc484215-kube-api-access-8dxcx\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.401637 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.401592    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.401676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.401636    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b34c2-4296-430e-82a0-c98fbc484215-openshift-service-ca\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.502407 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.502367    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b34c2-4296-430e-82a0-c98fbc484215-openshift-service-ca\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.502588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.502423    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-metrics-certs\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.502588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.502470    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxcx\" (UniqueName: \"kubernetes.io/projected/044b34c2-4296-430e-82a0-c98fbc484215-kube-api-access-8dxcx\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.502588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.502501    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.502734 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:42.502592    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-serving-cert: secret "maas-api-e2e-ait-aa5b913c-serving-cert" not found
Nov 02 17:33:42.502734 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:42.502649    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls podName:044b34c2-4296-430e-82a0-c98fbc484215 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:43.002634232 +0000 UTC m=+653.259664830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls") pod "maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq" (UID: "044b34c2-4296-430e-82a0-c98fbc484215") : secret "maas-api-e2e-ait-aa5b913c-serving-cert" not found
Nov 02 17:33:42.503211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.503189    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b34c2-4296-430e-82a0-c98fbc484215-openshift-service-ca\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.505130 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.505101    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-metrics-certs\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:42.560992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:42.560958    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxcx\" (UniqueName: \"kubernetes.io/projected/044b34c2-4296-430e-82a0-c98fbc484215-kube-api-access-8dxcx\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:43.006928 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:43.006893    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:43.009466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:43.009446    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls\") pod \"maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:43.240252 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:43.240219    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:43.613700 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:43.613676    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"]
Nov 02 17:33:43.616273 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:43.616240    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044b34c2_4296_430e_82a0_c98fbc484215.slice/crio-391d2b034daf8ee0716efc2ab3c9f003217c0c38317e178e56361a6ad2e38a41 WatchSource:0}: Error finding container 391d2b034daf8ee0716efc2ab3c9f003217c0c38317e178e56361a6ad2e38a41: Status 404 returned error can't find the container with id 391d2b034daf8ee0716efc2ab3c9f003217c0c38317e178e56361a6ad2e38a41
Nov 02 17:33:43.618055 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:43.618035    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:33:43.639191 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:43.639155    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq" event={"ID":"044b34c2-4296-430e-82a0-c98fbc484215","Type":"ContainerStarted","Data":"391d2b034daf8ee0716efc2ab3c9f003217c0c38317e178e56361a6ad2e38a41"}
Nov 02 17:33:44.645272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:44.645237    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq" event={"ID":"044b34c2-4296-430e-82a0-c98fbc484215","Type":"ContainerStarted","Data":"cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77"}
Nov 02 17:33:44.645744 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:44.645366    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:44.800304 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:44.800192    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq" podStartSLOduration=1.9072137279999999 podStartE2EDuration="2.800173001s" totalImagesPullingTime="892.959273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:44.798274765 +0000 UTC m=+655.055305380" watchObservedRunningTime="2026-11-02 17:33:44.800173001 +0000 UTC m=+655.057203619"
Nov 02 17:33:45.614402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.614370    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"]
Nov 02 17:33:45.618066 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.618050    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.674223 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.674186    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"]
Nov 02 17:33:45.720838 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.720810    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-97ba043b\""
Nov 02 17:33:45.721063 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.721044    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-97ba043b-dockercfg-5w9th\""
Nov 02 17:33:45.730840 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.730800    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qf9j\" (UniqueName: \"kubernetes.io/projected/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-kube-api-access-9qf9j\") pod \"payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.731152 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.730921    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.772365 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.772325    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"]
Nov 02 17:33:45.776851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.776829    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:45.823964 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.823933    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"]
Nov 02 17:33:45.832756 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.832728    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qf9j\" (UniqueName: \"kubernetes.io/projected/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-kube-api-access-9qf9j\") pod \"payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.832895 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.832767    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-plugins-config-volume\") pod \"payload-processing-e2e-ait-97ba043b-74f5944f87-stz26\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") " pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:45.832935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.832894    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.833006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.832985    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2864\" (UniqueName: \"kubernetes.io/projected/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-kube-api-access-x2864\") pod \"payload-processing-e2e-ait-97ba043b-74f5944f87-stz26\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") " pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:45.833609 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.833593    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.892390 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.892317    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qf9j\" (UniqueName: \"kubernetes.io/projected/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-kube-api-access-9qf9j\") pod \"payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.931157 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.931098    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:45.934575 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.934098    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-plugins-config-volume\") pod \"payload-processing-e2e-ait-97ba043b-74f5944f87-stz26\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") " pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:45.934575 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.934183    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2864\" (UniqueName: \"kubernetes.io/projected/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-kube-api-access-x2864\") pod \"payload-processing-e2e-ait-97ba043b-74f5944f87-stz26\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") " pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:45.935341 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.935316    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-plugins-config-volume\") pod \"payload-processing-e2e-ait-97ba043b-74f5944f87-stz26\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") " pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:45.996071 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:45.996030    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2864\" (UniqueName: \"kubernetes.io/projected/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-kube-api-access-x2864\") pod \"payload-processing-e2e-ait-97ba043b-74f5944f87-stz26\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") " pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:46.090074 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.090038    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:46.303398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.303373    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"]
Nov 02 17:33:46.305550 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:46.305524    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f3ade3_20d4_4332_a74c_9d52ac2154a3.slice/crio-fd310764631b50d834571a0aa1bed1e27b50c81b6a0d13d705f13a08bebf4b34 WatchSource:0}: Error finding container fd310764631b50d834571a0aa1bed1e27b50c81b6a0d13d705f13a08bebf4b34: Status 404 returned error can't find the container with id fd310764631b50d834571a0aa1bed1e27b50c81b6a0d13d705f13a08bebf4b34
Nov 02 17:33:46.466456 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.466431    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"]
Nov 02 17:33:46.468428 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:46.468406    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03a78b4_1b71_4872_b2a0_3cc0d899b52b.slice/crio-f10b20045ff4b2f1511f7574edba86bac260deee37150cdcd6c174e915ed1772 WatchSource:0}: Error finding container f10b20045ff4b2f1511f7574edba86bac260deee37150cdcd6c174e915ed1772: Status 404 returned error can't find the container with id f10b20045ff4b2f1511f7574edba86bac260deee37150cdcd6c174e915ed1772
Nov 02 17:33:46.655936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.655897    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26" event={"ID":"d03a78b4-1b71-4872-b2a0-3cc0d899b52b","Type":"ContainerStarted","Data":"4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a"}
Nov 02 17:33:46.656139 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.655958    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26" event={"ID":"d03a78b4-1b71-4872-b2a0-3cc0d899b52b","Type":"ContainerStarted","Data":"f10b20045ff4b2f1511f7574edba86bac260deee37150cdcd6c174e915ed1772"}
Nov 02 17:33:46.656139 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.655985    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:46.658695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.658669    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf" event={"ID":"a9f3ade3-20d4-4332-a74c-9d52ac2154a3","Type":"ContainerStarted","Data":"6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d"}
Nov 02 17:33:46.658876 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.658716    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf" event={"ID":"a9f3ade3-20d4-4332-a74c-9d52ac2154a3","Type":"ContainerStarted","Data":"fd310764631b50d834571a0aa1bed1e27b50c81b6a0d13d705f13a08bebf4b34"}
Nov 02 17:33:46.658876 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.658814    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:46.770217 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.770068    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26" podStartSLOduration=1.7700137310000001 podStartE2EDuration="1.770013731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:46.769888416 +0000 UTC m=+657.026919037" watchObservedRunningTime="2026-11-02 17:33:46.770013731 +0000 UTC m=+657.027044349"
Nov 02 17:33:46.885271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:46.885224    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf" podStartSLOduration=1.8852078190000001 podStartE2EDuration="1.885207819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:46.883450863 +0000 UTC m=+657.140481485" watchObservedRunningTime="2026-11-02 17:33:46.885207819 +0000 UTC m=+657.142238473"
Nov 02 17:33:47.657994 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.657952    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"]
Nov 02 17:33:47.661705 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.661682    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.713300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.713273    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cc12ac52-serving-cert\""
Nov 02 17:33:47.720277 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.720223    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"]
Nov 02 17:33:47.756240 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.756199    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.756433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.756303    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-metrics-certs\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.756494 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.756434    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e77ba6a-1761-4352-8745-e9e4f0086600-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.756494 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.756475    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7vht\" (UniqueName: \"kubernetes.io/projected/3e77ba6a-1761-4352-8745-e9e4f0086600-kube-api-access-b7vht\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.857653 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.857608    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.857653 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.857655    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-metrics-certs\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.858212 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.857687    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e77ba6a-1761-4352-8745-e9e4f0086600-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.858212 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.857709    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vht\" (UniqueName: \"kubernetes.io/projected/3e77ba6a-1761-4352-8745-e9e4f0086600-kube-api-access-b7vht\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.858212 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:47.857801    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-serving-cert: secret "maas-api-e2e-migrate-cc12ac52-serving-cert" not found
Nov 02 17:33:47.858212 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:47.857889    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls podName:3e77ba6a-1761-4352-8745-e9e4f0086600 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:48.357865458 +0000 UTC m=+658.614896061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls") pod "maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl" (UID: "3e77ba6a-1761-4352-8745-e9e4f0086600") : secret "maas-api-e2e-migrate-cc12ac52-serving-cert" not found
Nov 02 17:33:47.858741 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.858691    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e77ba6a-1761-4352-8745-e9e4f0086600-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.860651 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.860622    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-metrics-certs\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:47.914717 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:47.914642    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vht\" (UniqueName: \"kubernetes.io/projected/3e77ba6a-1761-4352-8745-e9e4f0086600-kube-api-access-b7vht\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:48.275627 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.275592    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"]
Nov 02 17:33:48.279317 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.279294    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:48.332574 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.332408    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aa5b913c\""
Nov 02 17:33:48.335526 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.335487    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"]
Nov 02 17:33:48.362804 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.362764    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:48.362984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.362829    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2mt\" (UniqueName: \"kubernetes.io/projected/7261bb5e-4a23-4f57-9d12-712ee96ea41c-kube-api-access-8d2mt\") pod \"payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") " pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:48.362984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.362894    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7261bb5e-4a23-4f57-9d12-712ee96ea41c-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") " pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:48.365777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.365746    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls\") pod \"maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:48.403288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.403251    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aa5b913c-dockercfg-wdqf4\""
Nov 02 17:33:48.464422 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.464384    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2mt\" (UniqueName: \"kubernetes.io/projected/7261bb5e-4a23-4f57-9d12-712ee96ea41c-kube-api-access-8d2mt\") pod \"payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") " pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:48.464575 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.464439    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7261bb5e-4a23-4f57-9d12-712ee96ea41c-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") " pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:48.465167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.465113    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7261bb5e-4a23-4f57-9d12-712ee96ea41c-plugins-config-volume\") pod \"payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") " pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:48.521652 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.521625    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2mt\" (UniqueName: \"kubernetes.io/projected/7261bb5e-4a23-4f57-9d12-712ee96ea41c-kube-api-access-8d2mt\") pod \"payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") " pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:48.575826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.575736    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:48.595192 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:48.595164    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:49.001458 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:49.001421    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"]
Nov 02 17:33:49.003010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:49.002987    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"]
Nov 02 17:33:49.040218 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:49.040174    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e77ba6a_1761_4352_8745_e9e4f0086600.slice/crio-e03a8c014c441963d2a30e4a25a9b64c4a53887122e6bccaa6a331e26ff8e581 WatchSource:0}: Error finding container e03a8c014c441963d2a30e4a25a9b64c4a53887122e6bccaa6a331e26ff8e581: Status 404 returned error can't find the container with id e03a8c014c441963d2a30e4a25a9b64c4a53887122e6bccaa6a331e26ff8e581
Nov 02 17:33:49.041087 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:49.041060    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7261bb5e_4a23_4f57_9d12_712ee96ea41c.slice/crio-abf4c3d58213a9417fa892cee063810b4092bed61dae75d2ae967ff7c099f495 WatchSource:0}: Error finding container abf4c3d58213a9417fa892cee063810b4092bed61dae75d2ae967ff7c099f495: Status 404 returned error can't find the container with id abf4c3d58213a9417fa892cee063810b4092bed61dae75d2ae967ff7c099f495
Nov 02 17:33:49.678550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:49.678517    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl" event={"ID":"3e77ba6a-1761-4352-8745-e9e4f0086600","Type":"ContainerStarted","Data":"e03a8c014c441963d2a30e4a25a9b64c4a53887122e6bccaa6a331e26ff8e581"}
Nov 02 17:33:49.680104 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:49.680075    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r" event={"ID":"7261bb5e-4a23-4f57-9d12-712ee96ea41c","Type":"ContainerStarted","Data":"03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d"}
Nov 02 17:33:49.680251 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:49.680151    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:33:49.680251 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:49.680174    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r" event={"ID":"7261bb5e-4a23-4f57-9d12-712ee96ea41c","Type":"ContainerStarted","Data":"abf4c3d58213a9417fa892cee063810b4092bed61dae75d2ae967ff7c099f495"}
Nov 02 17:33:49.805792 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:49.805737    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r" podStartSLOduration=1.805722193 podStartE2EDuration="1.805722193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:49.803928655 +0000 UTC m=+660.060959275" watchObservedRunningTime="2026-11-02 17:33:49.805722193 +0000 UTC m=+660.062752813"
Nov 02 17:33:50.655493 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:50.655463    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:33:50.686881 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:50.686842    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl" event={"ID":"3e77ba6a-1761-4352-8745-e9e4f0086600","Type":"ContainerStarted","Data":"fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd"}
Nov 02 17:33:50.687047 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:50.687027    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:50.890222 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:50.890161    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl" podStartSLOduration=3.073345884 podStartE2EDuration="3.890144523s" totalImagesPullingTime="816.798639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:50.888697177 +0000 UTC m=+661.145727866" watchObservedRunningTime="2026-11-02 17:33:50.890144523 +0000 UTC m=+661.147175140"
Nov 02 17:33:52.636329 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.636295    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:33:52.636812 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.636365    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:33:52.801557 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.801523    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"]
Nov 02 17:33:52.805416 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.805392    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:52.860944 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.860904    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3eb29e6e-serving-cert\""
Nov 02 17:33:52.862976 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.862951    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"]
Nov 02 17:33:52.908538 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.908452    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:52.908538 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.908513    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-metrics-certs\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:52.908711 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.908610    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6bl\" (UniqueName: \"kubernetes.io/projected/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-kube-api-access-ws6bl\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:52.908752 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:52.908719    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-openshift-service-ca\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.009862 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.009818    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-metrics-certs\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.010041 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.009880    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6bl\" (UniqueName: \"kubernetes.io/projected/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-kube-api-access-ws6bl\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.010041 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.009961    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-openshift-service-ca\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.010041 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.010028    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.010259 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:53.010224    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-serving-cert: secret "maas-api-e2e-ait-3eb29e6e-serving-cert" not found
Nov 02 17:33:53.010334 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:53.010320    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls podName:a8474a17-5dcf-4fb5-8528-c96d9ae5dad5 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:53.510297034 +0000 UTC m=+663.767327637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls") pod "maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz" (UID: "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5") : secret "maas-api-e2e-ait-3eb29e6e-serving-cert" not found
Nov 02 17:33:53.010693 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.010677    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-openshift-service-ca\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.012504 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.012486    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-metrics-certs\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.075843 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.075809    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6bl\" (UniqueName: \"kubernetes.io/projected/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-kube-api-access-ws6bl\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.515614 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.515576    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.518388 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.518363    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls\") pod \"maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:53.719272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:53.719237    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:54.096880 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:54.096858    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"]
Nov 02 17:33:54.099425 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:33:54.099400    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8474a17_5dcf_4fb5_8528_c96d9ae5dad5.slice/crio-3630ffb1d045f13d7d0f61df1c2024f49c01ea9eb8b7ca8a3fe4f0f3cd261b24 WatchSource:0}: Error finding container 3630ffb1d045f13d7d0f61df1c2024f49c01ea9eb8b7ca8a3fe4f0f3cd261b24: Status 404 returned error can't find the container with id 3630ffb1d045f13d7d0f61df1c2024f49c01ea9eb8b7ca8a3fe4f0f3cd261b24
Nov 02 17:33:54.708024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:54.707986    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz" event={"ID":"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5","Type":"ContainerStarted","Data":"3630ffb1d045f13d7d0f61df1c2024f49c01ea9eb8b7ca8a3fe4f0f3cd261b24"}
Nov 02 17:33:55.715283 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:55.715247    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz" event={"ID":"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5","Type":"ContainerStarted","Data":"7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25"}
Nov 02 17:33:55.715748 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:55.715341    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:33:55.834964 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:55.834907    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz" podStartSLOduration=3.066976669 podStartE2EDuration="3.834890712s" totalImagesPullingTime="767.914043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:55.832677028 +0000 UTC m=+666.089707649" watchObservedRunningTime="2026-11-02 17:33:55.834890712 +0000 UTC m=+666.091921377"
Nov 02 17:33:56.696797 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:56.696767    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:33:57.666472 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:57.666447    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:33:57.666924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:33:57.666517    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:33:59.052611 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:33:59.052575    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:34:00.690887 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:00.690813    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:34:01.726848 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:01.726817    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:34:02.983239 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:02.983201    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"]
Nov 02 17:34:02.986879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:02.986857    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.091048 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.091012    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-97ba043b-serving-cert\""
Nov 02 17:34:03.093062 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.093039    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"]
Nov 02 17:34:03.114836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.114797    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.115020 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.114969    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-openshift-service-ca\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.115020 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.114997    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-metrics-certs\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.115139 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.115041    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rttc\" (UniqueName: \"kubernetes.io/projected/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-kube-api-access-4rttc\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.216040 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.216000    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-openshift-service-ca\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.216040 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.216044    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-metrics-certs\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.216296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.216091    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rttc\" (UniqueName: \"kubernetes.io/projected/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-kube-api-access-4rttc\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.216296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.216170    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.216413 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:03.216386    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-serving-cert: secret "maas-api-e2e-ait-97ba043b-serving-cert" not found
Nov 02 17:34:03.216513 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:03.216476    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls podName:52ade6ff-fb24-4f4b-969e-938f36a2a0ff nodeName:}" failed. No retries permitted until 2026-11-02 17:34:03.716452862 +0000 UTC m=+673.973483465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls") pod "maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" (UID: "52ade6ff-fb24-4f4b-969e-938f36a2a0ff") : secret "maas-api-e2e-ait-97ba043b-serving-cert" not found
Nov 02 17:34:03.216824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.216799    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-openshift-service-ca\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.218713 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.218689    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-metrics-certs\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.275800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.275719    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rttc\" (UniqueName: \"kubernetes.io/projected/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-kube-api-access-4rttc\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.722105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.722062    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.724808 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.724784    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls\") pod \"maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:03.900012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:03.899974    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:04.277585 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:04.277553    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"]
Nov 02 17:34:04.278213 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:34:04.278176    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ade6ff_fb24_4f4b_969e_938f36a2a0ff.slice/crio-d814b16cc13aaa8ac6c0bf6952e2e7e241a8adb8d334d1a4dafbb65da2d430db WatchSource:0}: Error finding container d814b16cc13aaa8ac6c0bf6952e2e7e241a8adb8d334d1a4dafbb65da2d430db: Status 404 returned error can't find the container with id d814b16cc13aaa8ac6c0bf6952e2e7e241a8adb8d334d1a4dafbb65da2d430db
Nov 02 17:34:04.760718 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:04.760682    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" event={"ID":"52ade6ff-fb24-4f4b-969e-938f36a2a0ff","Type":"ContainerStarted","Data":"d814b16cc13aaa8ac6c0bf6952e2e7e241a8adb8d334d1a4dafbb65da2d430db"}
Nov 02 17:34:05.768309 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:05.768270    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" event={"ID":"52ade6ff-fb24-4f4b-969e-938f36a2a0ff","Type":"ContainerStarted","Data":"8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7"}
Nov 02 17:34:05.768775 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:05.768399    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:05.891509 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:05.891448    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" podStartSLOduration=3.104110128 podStartE2EDuration="3.891431911s" totalImagesPullingTime="787.321783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:05.890147571 +0000 UTC m=+676.147178191" watchObservedRunningTime="2026-11-02 17:34:05.891431911 +0000 UTC m=+676.148462570"
Nov 02 17:34:11.780034 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:11.780006    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:17.874489 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.874453    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"]
Nov 02 17:34:17.879346 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.879325    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.970400 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.970359    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.970606 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.970425    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc843f16-b45a-43bc-ac74-3314aa932330-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.970701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.970683    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpqx\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-kube-api-access-wvpqx\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.970766 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.970720    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.970866 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.970851    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.970930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.970881    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.970991 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.970971    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.971088 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.971062    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:17.971233 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:17.971201    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc843f16-b45a-43bc-ac74-3314aa932330-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.009620 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.009588    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-97ba043b-openshift-default-dockercfg-c9rrg\""
Nov 02 17:34:18.024202 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.024173    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"]
Nov 02 17:34:18.072671 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072635    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.072671 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072674    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc843f16-b45a-43bc-ac74-3314aa932330-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.072924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072707    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpqx\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-kube-api-access-wvpqx\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.072924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072731    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.072924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072773    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.072924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072797    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.072924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072853    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.072924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072899    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.073253 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.072930    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc843f16-b45a-43bc-ac74-3314aa932330-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.073626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.073304    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.073626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.073353    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.073626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.073494    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.073626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.073582    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.073834 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.073818    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc843f16-b45a-43bc-ac74-3314aa932330-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.075457 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.075435    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc843f16-b45a-43bc-ac74-3314aa932330-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.075758 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.075739    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.137482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.137410    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.138312 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.138287    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpqx\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-kube-api-access-wvpqx\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.192821 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.192786    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:18.628528 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.628498    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"]
Nov 02 17:34:18.629939 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:34:18.629907    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc843f16_b45a_43bc_ac74_3314aa932330.slice/crio-aac4d6a8ba90af82da18a5359becc24c333c20edb36a37ada59fb78ec96c25e5 WatchSource:0}: Error finding container aac4d6a8ba90af82da18a5359becc24c333c20edb36a37ada59fb78ec96c25e5: Status 404 returned error can't find the container with id aac4d6a8ba90af82da18a5359becc24c333c20edb36a37ada59fb78ec96c25e5
Nov 02 17:34:18.632307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.632276    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:18.632407 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.632357    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:18.632407 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.632401    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:18.835733 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.835703    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" event={"ID":"fc843f16-b45a-43bc-ac74-3314aa932330","Type":"ContainerStarted","Data":"4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2"}
Nov 02 17:34:18.835924 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.835760    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" event={"ID":"fc843f16-b45a-43bc-ac74-3314aa932330","Type":"ContainerStarted","Data":"aac4d6a8ba90af82da18a5359becc24c333c20edb36a37ada59fb78ec96c25e5"}
Nov 02 17:34:18.955102 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:18.955046    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podStartSLOduration=1.955029958 podStartE2EDuration="1.955029958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:18.953984548 +0000 UTC m=+689.211015182" watchObservedRunningTime="2026-11-02 17:34:18.955029958 +0000 UTC m=+689.212060578"
Nov 02 17:34:19.193929 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:19.193897    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:19.195241 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:19.195213    4956 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:34:19.195398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:19.195264    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:34:20.193312 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:20.193272    4956 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:34:20.193792 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:20.193351    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:34:21.194073 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:21.194031    4956 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:34:21.194553 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:21.194096    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:34:22.193473 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:22.193437    4956 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:34:22.193651 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:22.193517    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:34:23.194179 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:23.194108    4956 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:34:23.194607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:23.194211    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:34:24.194081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:24.194043    4956 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:34:24.194314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:24.194108    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:34:25.198578 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:25.198497    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:25.199050 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:25.198870    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:25.199740 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:25.199720    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:34:27.671272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:27.671239    4956 patch_prober.go:29] interesting pod/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:34:27.671683 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:27.671291    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:34:28.920863 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:28.920830    4956 generic.go:356] "Generic (PLEG): container finished" podID="787ce83b-686e-453e-90d5-61381a488a68" containerID="09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275" exitCode=0
Nov 02 17:34:28.921330 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:28.920897    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" event={"ID":"787ce83b-686e-453e-90d5-61381a488a68","Type":"ContainerDied","Data":"09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275"}
Nov 02 17:34:28.921330 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:28.921324    4956 scope.go:122] "RemoveContainer" containerID="09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275"
Nov 02 17:34:28.922624 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:28.922594    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:28.922750 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:28.922674    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:28.922750 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:28.922708    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:29.054468 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:29.054426    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:34:29.952967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:29.952923    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" event={"ID":"787ce83b-686e-453e-90d5-61381a488a68","Type":"ContainerStarted","Data":"3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7"}
Nov 02 17:34:30.473950 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:30.473891    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:34:30.474299 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:30.474275    4956 patch_prober.go:29] interesting pod/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 17:34:30.474416 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:30.474341    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 17:34:31.474347 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:31.474313    4956 patch_prober.go:29] interesting pod/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 17:34:31.474848 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:31.474367    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 17:34:32.474658 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:32.474621    4956 patch_prober.go:29] interesting pod/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 17:34:32.475054 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:32.474698    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 17:34:33.286692 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:33.286659    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"]
Nov 02 17:34:33.473952 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:33.473908    4956 patch_prober.go:29] interesting pod/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 17:34:33.474171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:33.473981    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 17:34:34.474082 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:34.474045    4956 patch_prober.go:29] interesting pod/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 17:34:34.474558 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:34.474134    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 17:34:35.477644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:35.477615    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:34:35.478075 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:35.477877    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:34:35.478615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:35.478595    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:34:38.287198 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.287144    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r" podUID="7261bb5e-4a23-4f57-9d12-712ee96ea41c" containerName="payload-processing" containerID="cri-o://03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d" gracePeriod=25
Nov 02 17:34:38.541553 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.541485    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:34:38.636814 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.636776    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7261bb5e-4a23-4f57-9d12-712ee96ea41c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7261bb5e-4a23-4f57-9d12-712ee96ea41c-plugins-config-volume\") pod \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") "
Nov 02 17:34:38.636993 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.636925    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7261bb5e-4a23-4f57-9d12-712ee96ea41c-kube-api-access-8d2mt\" (UniqueName: \"kubernetes.io/projected/7261bb5e-4a23-4f57-9d12-712ee96ea41c-kube-api-access-8d2mt\") pod \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\" (UID: \"7261bb5e-4a23-4f57-9d12-712ee96ea41c\") "
Nov 02 17:34:38.637308 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.637282    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7261bb5e-4a23-4f57-9d12-712ee96ea41c-plugins-config-volume" pod "7261bb5e-4a23-4f57-9d12-712ee96ea41c" (UID: "7261bb5e-4a23-4f57-9d12-712ee96ea41c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:38.639153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.639102    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7261bb5e-4a23-4f57-9d12-712ee96ea41c-kube-api-access-8d2mt" pod "7261bb5e-4a23-4f57-9d12-712ee96ea41c" (UID: "7261bb5e-4a23-4f57-9d12-712ee96ea41c"). InnerVolumeSpecName "kube-api-access-8d2mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:38.737816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.737787    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2mt\" (UniqueName: \"kubernetes.io/projected/7261bb5e-4a23-4f57-9d12-712ee96ea41c-kube-api-access-8d2mt\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:38.737816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:38.737814    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7261bb5e-4a23-4f57-9d12-712ee96ea41c-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:39.000734 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.000697    4956 generic.go:356] "Generic (PLEG): container finished" podID="7261bb5e-4a23-4f57-9d12-712ee96ea41c" containerID="03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d" exitCode=0
Nov 02 17:34:39.000910 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.000745    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"
Nov 02 17:34:39.000910 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.000784    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r" event={"ID":"7261bb5e-4a23-4f57-9d12-712ee96ea41c","Type":"ContainerDied","Data":"03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d"}
Nov 02 17:34:39.000910 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.000851    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r" event={"ID":"7261bb5e-4a23-4f57-9d12-712ee96ea41c","Type":"ContainerDied","Data":"abf4c3d58213a9417fa892cee063810b4092bed61dae75d2ae967ff7c099f495"}
Nov 02 17:34:39.000910 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.000879    4956 scope.go:122] "RemoveContainer" containerID="03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d"
Nov 02 17:34:39.010173 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.010156    4956 scope.go:122] "RemoveContainer" containerID="03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d"
Nov 02 17:34:39.010426 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:39.010407    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d\": container with ID starting with 03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d not found: ID does not exist" containerID="03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d"
Nov 02 17:34:39.010495 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.010440    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d"} err="failed to get container status \"03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d\": rpc error: code = NotFound desc = could not find container \"03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d\": container with ID starting with 03bb63d7c3bb91bbc27f08ac0f21380d31f68d3662d902c7021d7da11bed525d not found: ID does not exist"
Nov 02 17:34:39.166889 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.166823    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"]
Nov 02 17:34:39.169156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.169130    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aa5b913c-54b69cf789-6wp5r"]
Nov 02 17:34:39.863944 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:39.863912    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7261bb5e-4a23-4f57-9d12-712ee96ea41c" path="/var/lib/kubelet/pods/7261bb5e-4a23-4f57-9d12-712ee96ea41c/volumes"
Nov 02 17:34:41.280108 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:41.280075    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"]
Nov 02 17:34:41.389728 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:41.389694    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"]
Nov 02 17:34:43.717359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:43.717325    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"]
Nov 02 17:34:43.717791 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:43.717515    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq" podUID="044b34c2-4296-430e-82a0-c98fbc484215" containerName="maas-api" containerID="cri-o://cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77" gracePeriod=30
Nov 02 17:34:43.971174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:43.971092    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:34:44.028390 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.028359    4956 generic.go:356] "Generic (PLEG): container finished" podID="044b34c2-4296-430e-82a0-c98fbc484215" containerID="cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77" exitCode=0
Nov 02 17:34:44.028562 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.028411    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"
Nov 02 17:34:44.028562 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.028411    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq" event={"ID":"044b34c2-4296-430e-82a0-c98fbc484215","Type":"ContainerDied","Data":"cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77"}
Nov 02 17:34:44.028562 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.028542    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq" event={"ID":"044b34c2-4296-430e-82a0-c98fbc484215","Type":"ContainerDied","Data":"391d2b034daf8ee0716efc2ab3c9f003217c0c38317e178e56361a6ad2e38a41"}
Nov 02 17:34:44.028669 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.028570    4956 scope.go:122] "RemoveContainer" containerID="cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77"
Nov 02 17:34:44.037284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.037252    4956 scope.go:122] "RemoveContainer" containerID="cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77"
Nov 02 17:34:44.037551 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:44.037533    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77\": container with ID starting with cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77 not found: ID does not exist" containerID="cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77"
Nov 02 17:34:44.037629 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.037565    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77"} err="failed to get container status \"cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77\": rpc error: code = NotFound desc = could not find container \"cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77\": container with ID starting with cb891f857452c49ee74cfa0ab6f1c053f30aac3ffafa9b2fbd6b146e3d502e77 not found: ID does not exist"
Nov 02 17:34:44.097208 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.097169    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044b34c2-4296-430e-82a0-c98fbc484215-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b34c2-4296-430e-82a0-c98fbc484215-openshift-service-ca\") pod \"044b34c2-4296-430e-82a0-c98fbc484215\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") "
Nov 02 17:34:44.097396 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.097285    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls\") pod \"044b34c2-4296-430e-82a0-c98fbc484215\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") "
Nov 02 17:34:44.097396 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.097364    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044b34c2-4296-430e-82a0-c98fbc484215-kube-api-access-8dxcx\" (UniqueName: \"kubernetes.io/projected/044b34c2-4296-430e-82a0-c98fbc484215-kube-api-access-8dxcx\") pod \"044b34c2-4296-430e-82a0-c98fbc484215\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") "
Nov 02 17:34:44.097515 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.097476    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-metrics-certs\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-metrics-certs\") pod \"044b34c2-4296-430e-82a0-c98fbc484215\" (UID: \"044b34c2-4296-430e-82a0-c98fbc484215\") "
Nov 02 17:34:44.097623 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.097596    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044b34c2-4296-430e-82a0-c98fbc484215-openshift-service-ca" pod "044b34c2-4296-430e-82a0-c98fbc484215" (UID: "044b34c2-4296-430e-82a0-c98fbc484215"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:44.098080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.098055    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044b34c2-4296-430e-82a0-c98fbc484215-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:44.099930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.099908    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-metrics-certs" pod "044b34c2-4296-430e-82a0-c98fbc484215" (UID: "044b34c2-4296-430e-82a0-c98fbc484215"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:44.100056 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.100036    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls" pod "044b34c2-4296-430e-82a0-c98fbc484215" (UID: "044b34c2-4296-430e-82a0-c98fbc484215"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:44.100342 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.100320    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044b34c2-4296-430e-82a0-c98fbc484215-kube-api-access-8dxcx" pod "044b34c2-4296-430e-82a0-c98fbc484215" (UID: "044b34c2-4296-430e-82a0-c98fbc484215"). InnerVolumeSpecName "kube-api-access-8dxcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:44.199462 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.199414    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:44.199462 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.199464    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044b34c2-4296-430e-82a0-c98fbc484215-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:44.199677 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.199480    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dxcx\" (UniqueName: \"kubernetes.io/projected/044b34c2-4296-430e-82a0-c98fbc484215-kube-api-access-8dxcx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:44.448321 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.448292    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"]
Nov 02 17:34:44.452642 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.452613    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aa5b913c-8655bf46b5-hhszq"]
Nov 02 17:34:44.947842 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.947811    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"]
Nov 02 17:34:44.948311 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:44.948007    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl" podUID="3e77ba6a-1761-4352-8745-e9e4f0086600" containerName="maas-api" containerID="cri-o://fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd" gracePeriod=30
Nov 02 17:34:45.205548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.205517    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:34:45.309422 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.309387    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e77ba6a-1761-4352-8745-e9e4f0086600-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e77ba6a-1761-4352-8745-e9e4f0086600-openshift-service-ca\") pod \"3e77ba6a-1761-4352-8745-e9e4f0086600\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") "
Nov 02 17:34:45.309610 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.309440    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls\") pod \"3e77ba6a-1761-4352-8745-e9e4f0086600\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") "
Nov 02 17:34:45.309610 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.309600    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e77ba6a-1761-4352-8745-e9e4f0086600-kube-api-access-b7vht\" (UniqueName: \"kubernetes.io/projected/3e77ba6a-1761-4352-8745-e9e4f0086600-kube-api-access-b7vht\") pod \"3e77ba6a-1761-4352-8745-e9e4f0086600\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") "
Nov 02 17:34:45.309696 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.309634    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-metrics-certs\") pod \"3e77ba6a-1761-4352-8745-e9e4f0086600\" (UID: \"3e77ba6a-1761-4352-8745-e9e4f0086600\") "
Nov 02 17:34:45.309766 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.309738    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e77ba6a-1761-4352-8745-e9e4f0086600-openshift-service-ca" pod "3e77ba6a-1761-4352-8745-e9e4f0086600" (UID: "3e77ba6a-1761-4352-8745-e9e4f0086600"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:45.309986 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.309966    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e77ba6a-1761-4352-8745-e9e4f0086600-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:45.311710 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.311679    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls" pod "3e77ba6a-1761-4352-8745-e9e4f0086600" (UID: "3e77ba6a-1761-4352-8745-e9e4f0086600"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:45.311804 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.311775    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-metrics-certs" pod "3e77ba6a-1761-4352-8745-e9e4f0086600" (UID: "3e77ba6a-1761-4352-8745-e9e4f0086600"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:45.311902 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.311881    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e77ba6a-1761-4352-8745-e9e4f0086600-kube-api-access-b7vht" pod "3e77ba6a-1761-4352-8745-e9e4f0086600" (UID: "3e77ba6a-1761-4352-8745-e9e4f0086600"). InnerVolumeSpecName "kube-api-access-b7vht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:45.411309 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.411271    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:45.411309 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.411303    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e77ba6a-1761-4352-8745-e9e4f0086600-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:45.411309 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.411315    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7vht\" (UniqueName: \"kubernetes.io/projected/3e77ba6a-1761-4352-8745-e9e4f0086600-kube-api-access-b7vht\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:45.863921 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:45.863884    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044b34c2-4296-430e-82a0-c98fbc484215" path="/var/lib/kubelet/pods/044b34c2-4296-430e-82a0-c98fbc484215/volumes"
Nov 02 17:34:46.040874 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.040835    4956 generic.go:356] "Generic (PLEG): container finished" podID="3e77ba6a-1761-4352-8745-e9e4f0086600" containerID="fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd" exitCode=0
Nov 02 17:34:46.041368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.040893    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl" event={"ID":"3e77ba6a-1761-4352-8745-e9e4f0086600","Type":"ContainerDied","Data":"fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd"}
Nov 02 17:34:46.041368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.040903    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"
Nov 02 17:34:46.041368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.040946    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl" event={"ID":"3e77ba6a-1761-4352-8745-e9e4f0086600","Type":"ContainerDied","Data":"e03a8c014c441963d2a30e4a25a9b64c4a53887122e6bccaa6a331e26ff8e581"}
Nov 02 17:34:46.041368 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.040971    4956 scope.go:122] "RemoveContainer" containerID="fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd"
Nov 02 17:34:46.049893 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.049871    4956 scope.go:122] "RemoveContainer" containerID="fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd"
Nov 02 17:34:46.050169 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:46.050151    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd\": container with ID starting with fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd not found: ID does not exist" containerID="fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd"
Nov 02 17:34:46.050231 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.050182    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd"} err="failed to get container status \"fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd\": rpc error: code = NotFound desc = could not find container \"fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd\": container with ID starting with fb8322d78d2b301bb57ff43bcd6f1f0ef1c6d5ba22a67c26581700d8b68c12cd not found: ID does not exist"
Nov 02 17:34:46.203735 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.203699    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"]
Nov 02 17:34:46.253409 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.253372    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc12ac52-556ccf4cb7-rwkcl"]
Nov 02 17:34:46.281474 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.281411    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52" podUID="a98853fd-8696-41df-b917-6634ba970658" containerName="payload-processing" containerID="cri-o://6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437" gracePeriod=25
Nov 02 17:34:46.391033 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.390986    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2" podUID="131778ce-a76b-484d-b605-c9997e59ce59" containerName="payload-pre-processing" containerID="cri-o://a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38" gracePeriod=25
Nov 02 17:34:46.563181 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.563153    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:34:46.624457 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.624423    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98853fd-8696-41df-b917-6634ba970658-kube-api-access-7w5t6\" (UniqueName: \"kubernetes.io/projected/a98853fd-8696-41df-b917-6634ba970658-kube-api-access-7w5t6\") pod \"a98853fd-8696-41df-b917-6634ba970658\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") "
Nov 02 17:34:46.624634 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.624553    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98853fd-8696-41df-b917-6634ba970658-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98853fd-8696-41df-b917-6634ba970658-plugins-config-volume\") pod \"a98853fd-8696-41df-b917-6634ba970658\" (UID: \"a98853fd-8696-41df-b917-6634ba970658\") "
Nov 02 17:34:46.624934 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.624906    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98853fd-8696-41df-b917-6634ba970658-plugins-config-volume" pod "a98853fd-8696-41df-b917-6634ba970658" (UID: "a98853fd-8696-41df-b917-6634ba970658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:46.627311 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.627282    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98853fd-8696-41df-b917-6634ba970658-kube-api-access-7w5t6" pod "a98853fd-8696-41df-b917-6634ba970658" (UID: "a98853fd-8696-41df-b917-6634ba970658"). InnerVolumeSpecName "kube-api-access-7w5t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:46.662015 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.661991    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:34:46.725473 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.725382    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131778ce-a76b-484d-b605-c9997e59ce59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131778ce-a76b-484d-b605-c9997e59ce59-plugins-config-volume\") pod \"131778ce-a76b-484d-b605-c9997e59ce59\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") "
Nov 02 17:34:46.725473 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.725423    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131778ce-a76b-484d-b605-c9997e59ce59-kube-api-access-p52mk\" (UniqueName: \"kubernetes.io/projected/131778ce-a76b-484d-b605-c9997e59ce59-kube-api-access-p52mk\") pod \"131778ce-a76b-484d-b605-c9997e59ce59\" (UID: \"131778ce-a76b-484d-b605-c9997e59ce59\") "
Nov 02 17:34:46.725806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.725791    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5t6\" (UniqueName: \"kubernetes.io/projected/a98853fd-8696-41df-b917-6634ba970658-kube-api-access-7w5t6\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:46.725867 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.725809    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a98853fd-8696-41df-b917-6634ba970658-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:46.725867 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.725824    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131778ce-a76b-484d-b605-c9997e59ce59-plugins-config-volume" pod "131778ce-a76b-484d-b605-c9997e59ce59" (UID: "131778ce-a76b-484d-b605-c9997e59ce59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:46.728254 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.728225    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131778ce-a76b-484d-b605-c9997e59ce59-kube-api-access-p52mk" pod "131778ce-a76b-484d-b605-c9997e59ce59" (UID: "131778ce-a76b-484d-b605-c9997e59ce59"). InnerVolumeSpecName "kube-api-access-p52mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:46.826471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.826436    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131778ce-a76b-484d-b605-c9997e59ce59-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:46.826471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:46.826469    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52mk\" (UniqueName: \"kubernetes.io/projected/131778ce-a76b-484d-b605-c9997e59ce59-kube-api-access-p52mk\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:47.048070 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.047976    4956 generic.go:356] "Generic (PLEG): container finished" podID="a98853fd-8696-41df-b917-6634ba970658" containerID="6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437" exitCode=0
Nov 02 17:34:47.048070 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.048030    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"
Nov 02 17:34:47.048070 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.048056    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52" event={"ID":"a98853fd-8696-41df-b917-6634ba970658","Type":"ContainerDied","Data":"6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437"}
Nov 02 17:34:47.048719 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.048111    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52" event={"ID":"a98853fd-8696-41df-b917-6634ba970658","Type":"ContainerDied","Data":"06a11785e763af03801e32fde42f1a5e69c8b31d6a93e3a175debe3f3e370737"}
Nov 02 17:34:47.048719 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.048161    4956 scope.go:122] "RemoveContainer" containerID="6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437"
Nov 02 17:34:47.050687 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.050660    4956 generic.go:356] "Generic (PLEG): container finished" podID="131778ce-a76b-484d-b605-c9997e59ce59" containerID="a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38" exitCode=0
Nov 02 17:34:47.050816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.050724    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"
Nov 02 17:34:47.050816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.050769    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2" event={"ID":"131778ce-a76b-484d-b605-c9997e59ce59","Type":"ContainerDied","Data":"a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38"}
Nov 02 17:34:47.050914 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.050820    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2" event={"ID":"131778ce-a76b-484d-b605-c9997e59ce59","Type":"ContainerDied","Data":"f3ab0eaeb4d47113805d9ae36fe5c149d031683473b2914a8ee6daf70f53afba"}
Nov 02 17:34:47.057563 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.057540    4956 scope.go:122] "RemoveContainer" containerID="6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437"
Nov 02 17:34:47.057896 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:47.057868    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437\": container with ID starting with 6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437 not found: ID does not exist" containerID="6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437"
Nov 02 17:34:47.058001 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.057907    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437"} err="failed to get container status \"6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437\": rpc error: code = NotFound desc = could not find container \"6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437\": container with ID starting with 6e36ccc9cbc12dda912564a7264bbcbd29148719063910cd23d06ea3e0e8f437 not found: ID does not exist"
Nov 02 17:34:47.058001 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.057932    4956 scope.go:122] "RemoveContainer" containerID="a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38"
Nov 02 17:34:47.069770 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.069745    4956 scope.go:122] "RemoveContainer" containerID="a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38"
Nov 02 17:34:47.070094 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:47.070072    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38\": container with ID starting with a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38 not found: ID does not exist" containerID="a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38"
Nov 02 17:34:47.070227 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.070106    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38"} err="failed to get container status \"a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38\": rpc error: code = NotFound desc = could not find container \"a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38\": container with ID starting with a6c7362aebf511e7a9cdd4aa32a3a23162627d8a941cd8aeb14d8669d9d3bc38 not found: ID does not exist"
Nov 02 17:34:47.213734 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.213697    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"]
Nov 02 17:34:47.219249 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.219223    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc12ac52-f4c7c9-w57s2"]
Nov 02 17:34:47.335973 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.335882    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"]
Nov 02 17:34:47.337965 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.337941    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc12ac52-6869d89cdf-dkz52"]
Nov 02 17:34:47.689187 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689078    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"]
Nov 02 17:34:47.689545 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689526    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a98853fd-8696-41df-b917-6634ba970658" containerName="payload-processing"
Nov 02 17:34:47.689545 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689543    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98853fd-8696-41df-b917-6634ba970658" containerName="payload-processing"
Nov 02 17:34:47.689617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689570    4956 cpu_manager.go:414] "containerMap: removing container" podUID="044b34c2-4296-430e-82a0-c98fbc484215" containerName="maas-api"
Nov 02 17:34:47.689617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689575    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044b34c2-4296-430e-82a0-c98fbc484215" containerName="maas-api"
Nov 02 17:34:47.689617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689585    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3e77ba6a-1761-4352-8745-e9e4f0086600" containerName="maas-api"
Nov 02 17:34:47.689617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689590    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e77ba6a-1761-4352-8745-e9e4f0086600" containerName="maas-api"
Nov 02 17:34:47.689617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689602    4956 cpu_manager.go:414] "containerMap: removing container" podUID="7261bb5e-4a23-4f57-9d12-712ee96ea41c" containerName="payload-processing"
Nov 02 17:34:47.689617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689606    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7261bb5e-4a23-4f57-9d12-712ee96ea41c" containerName="payload-processing"
Nov 02 17:34:47.689617 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689616    4956 cpu_manager.go:414] "containerMap: removing container" podUID="131778ce-a76b-484d-b605-c9997e59ce59" containerName="payload-pre-processing"
Nov 02 17:34:47.689816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689621    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131778ce-a76b-484d-b605-c9997e59ce59" containerName="payload-pre-processing"
Nov 02 17:34:47.689816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689700    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="044b34c2-4296-430e-82a0-c98fbc484215" containerName="maas-api"
Nov 02 17:34:47.689816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689707    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e77ba6a-1761-4352-8745-e9e4f0086600" containerName="maas-api"
Nov 02 17:34:47.689816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689714    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="7261bb5e-4a23-4f57-9d12-712ee96ea41c" containerName="payload-processing"
Nov 02 17:34:47.689816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689721    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="131778ce-a76b-484d-b605-c9997e59ce59" containerName="payload-pre-processing"
Nov 02 17:34:47.689816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.689728    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98853fd-8696-41df-b917-6634ba970658" containerName="payload-processing"
Nov 02 17:34:47.694047 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.694029    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.736413 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.736375    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.736761 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.736718    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.737048 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.737017    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.737471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.737450    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tct5\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-kube-api-access-5tct5\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.737966 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.737921    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.738068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.738034    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.738282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.738108    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.738282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.738192    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.738282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.738266    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.748212 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.748188    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a099c3-openshift-default-dockercfg-68w8k\""
Nov 02 17:34:47.787696 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.787663    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"]
Nov 02 17:34:47.839254 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839221    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839266    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839296    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839319    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839351    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839378    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839407    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tct5\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-kube-api-access-5tct5\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839786 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839495    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839786 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839554    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839890 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839810    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839890 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839839    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839995 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839916    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.839995 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.839963    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.840285 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.840264    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.842081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.842053    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.842397 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.842375    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.864429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.864394    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131778ce-a76b-484d-b605-c9997e59ce59" path="/var/lib/kubelet/pods/131778ce-a76b-484d-b605-c9997e59ce59/volumes"
Nov 02 17:34:47.864799 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.864784    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e77ba6a-1761-4352-8745-e9e4f0086600" path="/var/lib/kubelet/pods/3e77ba6a-1761-4352-8745-e9e4f0086600/volumes"
Nov 02 17:34:47.865102 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.865087    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98853fd-8696-41df-b917-6634ba970658" path="/var/lib/kubelet/pods/a98853fd-8696-41df-b917-6634ba970658/volumes"
Nov 02 17:34:47.896451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.896420    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:47.896603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:47.896477    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tct5\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-kube-api-access-5tct5\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:48.007004 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:48.006959    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:48.438268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:48.438238    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"]
Nov 02 17:34:48.439969 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:34:48.439937    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9ba38f_17de_42f7_9ee2_addf0fa5efb3.slice/crio-de3650e130e3384368251ac5218699ad53ebb0061b3c0785f7c0298391581f95 WatchSource:0}: Error finding container de3650e130e3384368251ac5218699ad53ebb0061b3c0785f7c0298391581f95: Status 404 returned error can't find the container with id de3650e130e3384368251ac5218699ad53ebb0061b3c0785f7c0298391581f95
Nov 02 17:34:48.442325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:48.442295    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:48.442388 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:48.442359    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:48.442433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:48.442390    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:34:49.065532 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:49.065498    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" event={"ID":"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3","Type":"ContainerStarted","Data":"bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0"}
Nov 02 17:34:49.065702 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:49.065552    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" event={"ID":"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3","Type":"ContainerStarted","Data":"de3650e130e3384368251ac5218699ad53ebb0061b3c0785f7c0298391581f95"}
Nov 02 17:34:49.183868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:49.183811    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podStartSLOduration=2.183795675 podStartE2EDuration="2.183795675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:49.181486034 +0000 UTC m=+719.438516654" watchObservedRunningTime="2026-11-02 17:34:49.183795675 +0000 UTC m=+719.440826295"
Nov 02 17:34:50.007963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:50.007919    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:50.013770 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:50.013746    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:50.070534 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:50.070495    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:50.071569 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:50.071549    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:34:55.647837 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.647803    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"]
Nov 02 17:34:55.653052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.653032    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:55.724922 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.724887    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklqt\" (UniqueName: \"kubernetes.io/projected/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-kube-api-access-zklqt\") pod \"payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:55.725103 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.724937    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:55.755459 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.755428    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a099c3\""
Nov 02 17:34:55.755459 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.755448    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a099c3-dockercfg-gbk7h\""
Nov 02 17:34:55.757344 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.757324    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"]
Nov 02 17:34:55.825768 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.825734    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklqt\" (UniqueName: \"kubernetes.io/projected/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-kube-api-access-zklqt\") pod \"payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:55.825768 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.825769    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:55.826504 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.826484    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:55.883918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.883886    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklqt\" (UniqueName: \"kubernetes.io/projected/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-kube-api-access-zklqt\") pod \"payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:55.978971 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:55.978936    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:56.354747 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.354722    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"]
Nov 02 17:34:56.357193 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:34:56.357168    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb26302_c3fd_42a0_8ea6_2ce10dc1fcd5.slice/crio-18528e5f15bd0aaa57cf754ba76a21c090cb0787137cd0e1bbf53af809b315d1 WatchSource:0}: Error finding container 18528e5f15bd0aaa57cf754ba76a21c090cb0787137cd0e1bbf53af809b315d1: Status 404 returned error can't find the container with id 18528e5f15bd0aaa57cf754ba76a21c090cb0787137cd0e1bbf53af809b315d1
Nov 02 17:34:56.364693 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.364666    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"]
Nov 02 17:34:56.470686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.470650    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"]
Nov 02 17:34:56.623073 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.622990    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"]
Nov 02 17:34:56.623267 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.623234    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" podUID="52ade6ff-fb24-4f4b-969e-938f36a2a0ff" containerName="maas-api" containerID="cri-o://8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7" gracePeriod=30
Nov 02 17:34:56.774824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.774786    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" podUID="52ade6ff-fb24-4f4b-969e-938f36a2a0ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.52:8443/health\": dial tcp 10.132.0.52:8443: connect: connection refused"
Nov 02 17:34:56.868143 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.868102    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:56.939815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.939776    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-openshift-service-ca\") pod \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") "
Nov 02 17:34:56.939815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.939815    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-kube-api-access-4rttc\" (UniqueName: \"kubernetes.io/projected/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-kube-api-access-4rttc\") pod \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") "
Nov 02 17:34:56.940047 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.939843    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-metrics-certs\") pod \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") "
Nov 02 17:34:56.940047 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.939940    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls\") pod \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\" (UID: \"52ade6ff-fb24-4f4b-969e-938f36a2a0ff\") "
Nov 02 17:34:56.940342 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.940303    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-openshift-service-ca" pod "52ade6ff-fb24-4f4b-969e-938f36a2a0ff" (UID: "52ade6ff-fb24-4f4b-969e-938f36a2a0ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:56.940923 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.940898    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:56.942851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.942824    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-metrics-certs" pod "52ade6ff-fb24-4f4b-969e-938f36a2a0ff" (UID: "52ade6ff-fb24-4f4b-969e-938f36a2a0ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:56.942851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.942832    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-kube-api-access-4rttc" pod "52ade6ff-fb24-4f4b-969e-938f36a2a0ff" (UID: "52ade6ff-fb24-4f4b-969e-938f36a2a0ff"). InnerVolumeSpecName "kube-api-access-4rttc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:56.942973 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:56.942846    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls" pod "52ade6ff-fb24-4f4b-969e-938f36a2a0ff" (UID: "52ade6ff-fb24-4f4b-969e-938f36a2a0ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:57.042087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.042058    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:57.042087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.042089    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rttc\" (UniqueName: \"kubernetes.io/projected/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-kube-api-access-4rttc\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:57.042328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.042101    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ade6ff-fb24-4f4b-969e-938f36a2a0ff-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:57.106874 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.106837    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" event={"ID":"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5","Type":"ContainerStarted","Data":"09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963"}
Nov 02 17:34:57.107046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.106900    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" event={"ID":"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5","Type":"ContainerStarted","Data":"18528e5f15bd0aaa57cf754ba76a21c090cb0787137cd0e1bbf53af809b315d1"}
Nov 02 17:34:57.107046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.107017    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:34:57.108029 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.108005    4956 generic.go:356] "Generic (PLEG): container finished" podID="52ade6ff-fb24-4f4b-969e-938f36a2a0ff" containerID="8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7" exitCode=0
Nov 02 17:34:57.108195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.108036    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" event={"ID":"52ade6ff-fb24-4f4b-969e-938f36a2a0ff","Type":"ContainerDied","Data":"8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7"}
Nov 02 17:34:57.108195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.108056    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"
Nov 02 17:34:57.108195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.108064    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg" event={"ID":"52ade6ff-fb24-4f4b-969e-938f36a2a0ff","Type":"ContainerDied","Data":"d814b16cc13aaa8ac6c0bf6952e2e7e241a8adb8d334d1a4dafbb65da2d430db"}
Nov 02 17:34:57.108195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.108085    4956 scope.go:122] "RemoveContainer" containerID="8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7"
Nov 02 17:34:57.120082 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.120064    4956 scope.go:122] "RemoveContainer" containerID="8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7"
Nov 02 17:34:57.120430 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:57.120403    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7\": container with ID starting with 8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7 not found: ID does not exist" containerID="8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7"
Nov 02 17:34:57.120513 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.120439    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7"} err="failed to get container status \"8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7\": rpc error: code = NotFound desc = could not find container \"8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7\": container with ID starting with 8cd8a81d943ef25e1eb015b3eafbdd95ead490601959468782feac9a2ee827f7 not found: ID does not exist"
Nov 02 17:34:57.222065 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.222013    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" podStartSLOduration=2.221995755 podStartE2EDuration="2.221995755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:57.22097374 +0000 UTC m=+727.478004361" watchObservedRunningTime="2026-11-02 17:34:57.221995755 +0000 UTC m=+727.479026384"
Nov 02 17:34:57.332149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.332101    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"]
Nov 02 17:34:57.339032 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.338480    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97ba043b-845c4687fb-2pvwg"]
Nov 02 17:34:57.863881 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:34:57.863849    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ade6ff-fb24-4f4b-969e-938f36a2a0ff" path="/var/lib/kubelet/pods/52ade6ff-fb24-4f4b-969e-938f36a2a0ff/volumes"
Nov 02 17:34:59.052958 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:34:59.052922    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:35:01.365091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.365052    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26" podUID="d03a78b4-1b71-4872-b2a0-3cc0d899b52b" containerName="payload-processing" containerID="cri-o://4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a" gracePeriod=25
Nov 02 17:35:01.471526 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.471485    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf" podUID="a9f3ade3-20d4-4332-a74c-9d52ac2154a3" containerName="payload-pre-processing" containerID="cri-o://6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d" gracePeriod=25
Nov 02 17:35:01.623854 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.623783    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:35:01.734019 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.733997    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:35:01.786365 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.786332    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-plugins-config-volume\") pod \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") "
Nov 02 17:35:01.786526 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.786409    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-kube-api-access-x2864\" (UniqueName: \"kubernetes.io/projected/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-kube-api-access-x2864\") pod \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\" (UID: \"d03a78b4-1b71-4872-b2a0-3cc0d899b52b\") "
Nov 02 17:35:01.786689 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.786668    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-plugins-config-volume" pod "d03a78b4-1b71-4872-b2a0-3cc0d899b52b" (UID: "d03a78b4-1b71-4872-b2a0-3cc0d899b52b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:01.788543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.788524    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-kube-api-access-x2864" pod "d03a78b4-1b71-4872-b2a0-3cc0d899b52b" (UID: "d03a78b4-1b71-4872-b2a0-3cc0d899b52b"). InnerVolumeSpecName "kube-api-access-x2864". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:01.887982 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.887911    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-kube-api-access-9qf9j\" (UniqueName: \"kubernetes.io/projected/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-kube-api-access-9qf9j\") pod \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") "
Nov 02 17:35:01.888144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.888068    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-plugins-config-volume\") pod \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\" (UID: \"a9f3ade3-20d4-4332-a74c-9d52ac2154a3\") "
Nov 02 17:35:01.888275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.888261    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2864\" (UniqueName: \"kubernetes.io/projected/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-kube-api-access-x2864\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:01.888355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.888275    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03a78b4-1b71-4872-b2a0-3cc0d899b52b-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:01.888445 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.888422    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-plugins-config-volume" pod "a9f3ade3-20d4-4332-a74c-9d52ac2154a3" (UID: "a9f3ade3-20d4-4332-a74c-9d52ac2154a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:01.890220 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.890199    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-kube-api-access-9qf9j" pod "a9f3ade3-20d4-4332-a74c-9d52ac2154a3" (UID: "a9f3ade3-20d4-4332-a74c-9d52ac2154a3"). InnerVolumeSpecName "kube-api-access-9qf9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:01.989823 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.989773    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qf9j\" (UniqueName: \"kubernetes.io/projected/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-kube-api-access-9qf9j\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:01.989823 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:01.989818    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f3ade3-20d4-4332-a74c-9d52ac2154a3-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:02.133629 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.133592    4956 generic.go:356] "Generic (PLEG): container finished" podID="d03a78b4-1b71-4872-b2a0-3cc0d899b52b" containerID="4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a" exitCode=0
Nov 02 17:35:02.133807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.133656    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"
Nov 02 17:35:02.133807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.133669    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26" event={"ID":"d03a78b4-1b71-4872-b2a0-3cc0d899b52b","Type":"ContainerDied","Data":"4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a"}
Nov 02 17:35:02.133807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.133753    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26" event={"ID":"d03a78b4-1b71-4872-b2a0-3cc0d899b52b","Type":"ContainerDied","Data":"f10b20045ff4b2f1511f7574edba86bac260deee37150cdcd6c174e915ed1772"}
Nov 02 17:35:02.133807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.133781    4956 scope.go:122] "RemoveContainer" containerID="4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a"
Nov 02 17:35:02.134974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.134947    4956 generic.go:356] "Generic (PLEG): container finished" podID="a9f3ade3-20d4-4332-a74c-9d52ac2154a3" containerID="6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d" exitCode=0
Nov 02 17:35:02.135104 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.134994    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"
Nov 02 17:35:02.135104 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.134994    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf" event={"ID":"a9f3ade3-20d4-4332-a74c-9d52ac2154a3","Type":"ContainerDied","Data":"6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d"}
Nov 02 17:35:02.135252 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.135109    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf" event={"ID":"a9f3ade3-20d4-4332-a74c-9d52ac2154a3","Type":"ContainerDied","Data":"fd310764631b50d834571a0aa1bed1e27b50c81b6a0d13d705f13a08bebf4b34"}
Nov 02 17:35:02.143144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.143110    4956 scope.go:122] "RemoveContainer" containerID="4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a"
Nov 02 17:35:02.143493 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:02.143471    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a\": container with ID starting with 4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a not found: ID does not exist" containerID="4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a"
Nov 02 17:35:02.143551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.143503    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a"} err="failed to get container status \"4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a\": rpc error: code = NotFound desc = could not find container \"4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a\": container with ID starting with 4ddff793c6463001a1aa086932179a938edaf2d9a5e77bc7464f6d9150f0044a not found: ID does not exist"
Nov 02 17:35:02.143551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.143520    4956 scope.go:122] "RemoveContainer" containerID="6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d"
Nov 02 17:35:02.151832 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.151812    4956 scope.go:122] "RemoveContainer" containerID="6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d"
Nov 02 17:35:02.152108 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:02.152088    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d\": container with ID starting with 6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d not found: ID does not exist" containerID="6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d"
Nov 02 17:35:02.152180 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.152142    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d"} err="failed to get container status \"6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d\": rpc error: code = NotFound desc = could not find container \"6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d\": container with ID starting with 6a1e3002a78003235226c1ade8fcac0aae5c5e60a61dc3186d54389f4f3cd81d not found: ID does not exist"
Nov 02 17:35:02.299862 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.299825    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"]
Nov 02 17:35:02.302841 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.302817    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-97ba043b-6bc98f799f-57jrf"]
Nov 02 17:35:02.413243 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.413149    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"]
Nov 02 17:35:02.415597 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:02.415571    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97ba043b-74f5944f87-stz26"]
Nov 02 17:35:03.037984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.037947    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"]
Nov 02 17:35:03.038194 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.038173    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz" podUID="a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" containerName="maas-api" containerID="cri-o://7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25" gracePeriod=30
Nov 02 17:35:03.194777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.194741    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"]
Nov 02 17:35:03.195229 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195212    4956 cpu_manager.go:414] "containerMap: removing container" podUID="52ade6ff-fb24-4f4b-969e-938f36a2a0ff" containerName="maas-api"
Nov 02 17:35:03.195296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195232    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ade6ff-fb24-4f4b-969e-938f36a2a0ff" containerName="maas-api"
Nov 02 17:35:03.195296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195272    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a9f3ade3-20d4-4332-a74c-9d52ac2154a3" containerName="payload-pre-processing"
Nov 02 17:35:03.195296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195281    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f3ade3-20d4-4332-a74c-9d52ac2154a3" containerName="payload-pre-processing"
Nov 02 17:35:03.195411 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195301    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d03a78b4-1b71-4872-b2a0-3cc0d899b52b" containerName="payload-processing"
Nov 02 17:35:03.195411 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195309    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03a78b4-1b71-4872-b2a0-3cc0d899b52b" containerName="payload-processing"
Nov 02 17:35:03.195483 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195413    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ade6ff-fb24-4f4b-969e-938f36a2a0ff" containerName="maas-api"
Nov 02 17:35:03.195483 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195431    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f3ade3-20d4-4332-a74c-9d52ac2154a3" containerName="payload-pre-processing"
Nov 02 17:35:03.195483 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.195442    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03a78b4-1b71-4872-b2a0-3cc0d899b52b" containerName="payload-processing"
Nov 02 17:35:03.199234 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.199209    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.232343 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.232315    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"]
Nov 02 17:35:03.236559 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.236536    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.261534 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.261503    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"]
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310450    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310537    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310624    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310663    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310807    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27673e46-a4c9-472a-a222-a535807d7cd0-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310864    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310919    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.310978    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27673e46-a4c9-472a-a222-a535807d7cd0-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.311089    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67mlk\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-kube-api-access-67mlk\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.314140    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"]
Nov 02 17:35:03.326667 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.326640    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:35:03.412632 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412592    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.412800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412640    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.412800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412659    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27673e46-a4c9-472a-a222-a535807d7cd0-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.412800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412692    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67mlk\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-kube-api-access-67mlk\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.412800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412758    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.412800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412783    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.413092 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412843    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.413092 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.412963    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.413092 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413018    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.413294 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413111    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9057858f-a22e-4075-b45b-776a6b468386-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.413294 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413210    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.413294 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413234    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.413794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413312    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.413794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413384    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.413794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413444    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.413794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413477    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlh2\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-kube-api-access-bhlh2\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.413794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413542    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9057858f-a22e-4075-b45b-776a6b468386-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.413794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413778    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.414155 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.413963    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.415154 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.414394    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27673e46-a4c9-472a-a222-a535807d7cd0-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.415154 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.414440    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.415154 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.414785    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.415154 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.415074    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27673e46-a4c9-472a-a222-a535807d7cd0-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.415951 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.415930    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27673e46-a4c9-472a-a222-a535807d7cd0-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.416280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.416255    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.470064 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.470033    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.470228 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.470101    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67mlk\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-kube-api-access-67mlk\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.514504 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.514472    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:03.515676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.515652    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-kube-api-access-ws6bl\" (UniqueName: \"kubernetes.io/projected/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-kube-api-access-ws6bl\") pod \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") "
Nov 02 17:35:03.515805 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.515766    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-openshift-service-ca\") pod \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") "
Nov 02 17:35:03.515805 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.515796    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls\") pod \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") "
Nov 02 17:35:03.515918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.515826    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-metrics-certs\") pod \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\" (UID: \"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5\") "
Nov 02 17:35:03.515983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.515920    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516032    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516081    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516221 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516148    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516221 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516165    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-openshift-service-ca" pod "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" (UID: "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:03.516328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516224    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516300    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9057858f-a22e-4075-b45b-776a6b468386-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516428 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516352    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516347    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516612    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516619    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516903 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516772    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlh2\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-kube-api-access-bhlh2\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516903 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516823    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9057858f-a22e-4075-b45b-776a6b468386-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.516903 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516864    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.517068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.516929    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:03.517393 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.517369    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9057858f-a22e-4075-b45b-776a6b468386-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.518589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.518407    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls" pod "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" (UID: "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:35:03.518589 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.518517    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-metrics-certs" pod "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" (UID: "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:35:03.518895 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.518745    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-kube-api-access-ws6bl" pod "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" (UID: "a8474a17-5dcf-4fb5-8528-c96d9ae5dad5"). InnerVolumeSpecName "kube-api-access-ws6bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:03.519341 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.519321    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.519451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.519394    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9057858f-a22e-4075-b45b-776a6b468386-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.575174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.575090    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.576476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.576447    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlh2\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-kube-api-access-bhlh2\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.617676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.617643    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:03.617676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.617673    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6bl\" (UniqueName: \"kubernetes.io/projected/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-kube-api-access-ws6bl\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:03.617676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.617685    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:03.868371 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.868260    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:03.870855 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.870824    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f3ade3-20d4-4332-a74c-9d52ac2154a3" path="/var/lib/kubelet/pods/a9f3ade3-20d4-4332-a74c-9d52ac2154a3/volumes"
Nov 02 17:35:03.871276 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.871260    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03a78b4-1b71-4872-b2a0-3cc0d899b52b" path="/var/lib/kubelet/pods/d03a78b4-1b71-4872-b2a0-3cc0d899b52b/volumes"
Nov 02 17:35:03.944206 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.944140    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"]
Nov 02 17:35:03.946673 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:03.946647    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27673e46_a4c9_472a_a222_a535807d7cd0.slice/crio-65b820b7d7591802ef21ec9b454e79e4ccee463b8d9125ad6a450f59ef6b8edf WatchSource:0}: Error finding container 65b820b7d7591802ef21ec9b454e79e4ccee463b8d9125ad6a450f59ef6b8edf: Status 404 returned error can't find the container with id 65b820b7d7591802ef21ec9b454e79e4ccee463b8d9125ad6a450f59ef6b8edf
Nov 02 17:35:03.949168 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.949132    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:03.949260 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.949222    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:03.949311 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:03.949266    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:04.150103 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.150017    4956 generic.go:356] "Generic (PLEG): container finished" podID="a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" containerID="7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25" exitCode=0
Nov 02 17:35:04.150103 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.150098    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"
Nov 02 17:35:04.150324 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.150096    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz" event={"ID":"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5","Type":"ContainerDied","Data":"7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25"}
Nov 02 17:35:04.150324 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.150162    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz" event={"ID":"a8474a17-5dcf-4fb5-8528-c96d9ae5dad5","Type":"ContainerDied","Data":"3630ffb1d045f13d7d0f61df1c2024f49c01ea9eb8b7ca8a3fe4f0f3cd261b24"}
Nov 02 17:35:04.150324 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.150184    4956 scope.go:122] "RemoveContainer" containerID="7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25"
Nov 02 17:35:04.155341 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.155316    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx" event={"ID":"27673e46-a4c9-472a-a222-a535807d7cd0","Type":"ContainerStarted","Data":"b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f"}
Nov 02 17:35:04.155460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.155357    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx" event={"ID":"27673e46-a4c9-472a-a222-a535807d7cd0","Type":"ContainerStarted","Data":"65b820b7d7591802ef21ec9b454e79e4ccee463b8d9125ad6a450f59ef6b8edf"}
Nov 02 17:35:04.159890 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.159871    4956 scope.go:122] "RemoveContainer" containerID="7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25"
Nov 02 17:35:04.160186 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:04.160167    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25\": container with ID starting with 7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25 not found: ID does not exist" containerID="7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25"
Nov 02 17:35:04.160255 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.160196    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25"} err="failed to get container status \"7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25\": rpc error: code = NotFound desc = could not find container \"7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25\": container with ID starting with 7509867d5a385b7eea525dd809128ed5b72c522cff39047a8b3283aa69ea1f25 not found: ID does not exist"
Nov 02 17:35:04.243810 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.243788    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"]
Nov 02 17:35:04.245747 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:04.245717    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9057858f_a22e_4075_b45b_776a6b468386.slice/crio-665d1ac82f77575a7da8917b5a857019988f99f8e0d782ddce87bdf02fe65c2c WatchSource:0}: Error finding container 665d1ac82f77575a7da8917b5a857019988f99f8e0d782ddce87bdf02fe65c2c: Status 404 returned error can't find the container with id 665d1ac82f77575a7da8917b5a857019988f99f8e0d782ddce87bdf02fe65c2c
Nov 02 17:35:04.248156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.248096    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:04.248258 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.248188    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:04.248258 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.248218    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:04.313529 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.313499    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"]
Nov 02 17:35:04.317399 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.317373    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eb29e6e-744bddd75f-spgcz"]
Nov 02 17:35:04.432836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.432741    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx" podStartSLOduration=1.432725133 podStartE2EDuration="1.432725133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:04.429909198 +0000 UTC m=+734.686939823" watchObservedRunningTime="2026-11-02 17:35:04.432725133 +0000 UTC m=+734.689755886"
Nov 02 17:35:04.514905 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.514865    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:04.578442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.578411    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"]
Nov 02 17:35:04.578915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.578902    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" containerName="maas-api"
Nov 02 17:35:04.578957 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.578920    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" containerName="maas-api"
Nov 02 17:35:04.578998 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.578988    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" containerName="maas-api"
Nov 02 17:35:04.582225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.582210    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.626917 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.626881    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.627112 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.626934    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-metrics-certs\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.627112 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.627087    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.627277 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.627153    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttq72\" (UniqueName: \"kubernetes.io/projected/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-kube-api-access-ttq72\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.634330 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.634309    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-a099c3-serving-cert\""
Nov 02 17:35:04.640954 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.640926    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"]
Nov 02 17:35:04.728280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.728246    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-metrics-certs\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.728479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.728311    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.728479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.728345    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttq72\" (UniqueName: \"kubernetes.io/projected/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-kube-api-access-ttq72\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.728479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.728453    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.728649 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:04.728473    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-serving-cert: secret "maas-api-e2e-worker-w2-a099c3-serving-cert" not found
Nov 02 17:35:04.728649 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:04.728559    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls podName:ed6e2db6-5a36-4c13-8fde-1ccb7db50de4 nodeName:}" failed. No retries permitted until 2026-11-02 17:35:05.228523742 +0000 UTC m=+735.485554340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls") pod "maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26" (UID: "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4") : secret "maas-api-e2e-worker-w2-a099c3-serving-cert" not found
Nov 02 17:35:04.729163 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.729140    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.731170 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.731145    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-metrics-certs\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:04.788305 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:04.788274    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttq72\" (UniqueName: \"kubernetes.io/projected/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-kube-api-access-ttq72\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:05.162153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.162036    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw" event={"ID":"9057858f-a22e-4075-b45b-776a6b468386","Type":"ContainerStarted","Data":"b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b"}
Nov 02 17:35:05.162153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.162104    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw" event={"ID":"9057858f-a22e-4075-b45b-776a6b468386","Type":"ContainerStarted","Data":"665d1ac82f77575a7da8917b5a857019988f99f8e0d782ddce87bdf02fe65c2c"}
Nov 02 17:35:05.234508 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.234469    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:05.237303 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.237273    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:05.287042 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.286992    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw" podStartSLOduration=2.286976825 podStartE2EDuration="2.286976825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:05.285390264 +0000 UTC m=+735.542420884" watchObservedRunningTime="2026-11-02 17:35:05.286976825 +0000 UTC m=+735.544007444"
Nov 02 17:35:05.338356 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.338322    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"]
Nov 02 17:35:05.342801 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.342775    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:05.395369 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.395338    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6e3175\""
Nov 02 17:35:05.398245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.398216    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"]
Nov 02 17:35:05.461572 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.461535    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-6e3175-dockercfg-xddj7\""
Nov 02 17:35:05.471044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.471013    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"]
Nov 02 17:35:05.474650 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.474634    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.495687 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.495661    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:05.520444 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.520416    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:05.533625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.533591    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"]
Nov 02 17:35:05.539709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.539480    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869c0472-aebf-4527-a67b-703dd58994c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.539709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.539588    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:05.539709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.539645    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cnq\" (UniqueName: \"kubernetes.io/projected/869c0472-aebf-4527-a67b-703dd58994c0-kube-api-access-p2cnq\") pod \"payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.539709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.539666    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgrm\" (UniqueName: \"kubernetes.io/projected/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-kube-api-access-pfgrm\") pod \"payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:05.640832 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.640792    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cnq\" (UniqueName: \"kubernetes.io/projected/869c0472-aebf-4527-a67b-703dd58994c0-kube-api-access-p2cnq\") pod \"payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.640832 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.640836    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgrm\" (UniqueName: \"kubernetes.io/projected/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-kube-api-access-pfgrm\") pod \"payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:05.641084 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.640934    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869c0472-aebf-4527-a67b-703dd58994c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.641084 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.640983    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:05.641722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.641697    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:05.641834 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.641813    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869c0472-aebf-4527-a67b-703dd58994c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.704850 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.704821    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgrm\" (UniqueName: \"kubernetes.io/projected/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-kube-api-access-pfgrm\") pod \"payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:05.707674 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.707648    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cnq\" (UniqueName: \"kubernetes.io/projected/869c0472-aebf-4527-a67b-703dd58994c0-kube-api-access-p2cnq\") pod \"payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.787488 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.787397    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:05.866316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.866278    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8474a17-5dcf-4fb5-8528-c96d9ae5dad5" path="/var/lib/kubelet/pods/a8474a17-5dcf-4fb5-8528-c96d9ae5dad5/volumes"
Nov 02 17:35:05.868582 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.868560    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:05.873680 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.873657    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:05.888554 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:05.888526    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6e2db6_5a36_4c13_8fde_1ccb7db50de4.slice/crio-a2c2856e82dc948f1ff25329decf51bebdd909cecfe8a579c542e9cc50a4c76c WatchSource:0}: Error finding container a2c2856e82dc948f1ff25329decf51bebdd909cecfe8a579c542e9cc50a4c76c: Status 404 returned error can't find the container with id a2c2856e82dc948f1ff25329decf51bebdd909cecfe8a579c542e9cc50a4c76c
Nov 02 17:35:05.889350 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.889313    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"]
Nov 02 17:35:05.957512 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:05.957483    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:06.169230 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:06.169093    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26" event={"ID":"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4","Type":"ContainerStarted","Data":"a2c2856e82dc948f1ff25329decf51bebdd909cecfe8a579c542e9cc50a4c76c"}
Nov 02 17:35:06.169592 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:06.169565    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:06.169720 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:06.169623    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:06.170477 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:06.170452    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:06.170818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:06.170804    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:06.185706 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:06.185684    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"]
Nov 02 17:35:06.186869 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:06.186849    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869c0472_aebf_4527_a67b_703dd58994c0.slice/crio-fb3edf119bf9a60668b8f77a2001d6eb9a63ba622d3256a628da7b153d87a592 WatchSource:0}: Error finding container fb3edf119bf9a60668b8f77a2001d6eb9a63ba622d3256a628da7b153d87a592: Status 404 returned error can't find the container with id fb3edf119bf9a60668b8f77a2001d6eb9a63ba622d3256a628da7b153d87a592
Nov 02 17:35:06.333960 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:06.333935    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"]
Nov 02 17:35:06.335423 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:06.335397    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0601d8b3_f4a6_4680_b5d6_a0b3b1d7daa8.slice/crio-a2ec98de314de5f1c40cd722775e39e0a18dfd1d4c02bc214fef645fafb1adc9 WatchSource:0}: Error finding container a2ec98de314de5f1c40cd722775e39e0a18dfd1d4c02bc214fef645fafb1adc9: Status 404 returned error can't find the container with id a2ec98de314de5f1c40cd722775e39e0a18dfd1d4c02bc214fef645fafb1adc9
Nov 02 17:35:07.175688 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.175591    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" event={"ID":"869c0472-aebf-4527-a67b-703dd58994c0","Type":"ContainerStarted","Data":"ad8db7a6ec475ae95e048c3744b70dadb1e894c3f8d0f76d50164a1c323daede"}
Nov 02 17:35:07.175688 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.175653    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" event={"ID":"869c0472-aebf-4527-a67b-703dd58994c0","Type":"ContainerStarted","Data":"fb3edf119bf9a60668b8f77a2001d6eb9a63ba622d3256a628da7b153d87a592"}
Nov 02 17:35:07.176245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.175775    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:07.177547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.177522    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" event={"ID":"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8","Type":"ContainerStarted","Data":"ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf"}
Nov 02 17:35:07.177677 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.177565    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" event={"ID":"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8","Type":"ContainerStarted","Data":"a2ec98de314de5f1c40cd722775e39e0a18dfd1d4c02bc214fef645fafb1adc9"}
Nov 02 17:35:07.177677 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.177587    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:07.179155 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.179108    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26" event={"ID":"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4","Type":"ContainerStarted","Data":"90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550"}
Nov 02 17:35:07.291868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.291817    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" podStartSLOduration=2.291801912 podStartE2EDuration="2.291801912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:07.28985422 +0000 UTC m=+737.546884840" watchObservedRunningTime="2026-11-02 17:35:07.291801912 +0000 UTC m=+737.548832563"
Nov 02 17:35:07.413020 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.412968    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26" podStartSLOduration=2.516953471 podStartE2EDuration="3.412954595s" totalImagesPullingTime="896.001124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:07.411316799 +0000 UTC m=+737.668347426" watchObservedRunningTime="2026-11-02 17:35:07.412954595 +0000 UTC m=+737.669985214"
Nov 02 17:35:07.529783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:07.529723    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" podStartSLOduration=2.529705972 podStartE2EDuration="2.529705972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:07.526876748 +0000 UTC m=+737.783907369" watchObservedRunningTime="2026-11-02 17:35:07.529705972 +0000 UTC m=+737.786736602"
Nov 02 17:35:08.117609 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:08.117576    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 17:35:08.184266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:08.184233    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:10.490854 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:10.490818    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"]
Nov 02 17:35:10.491272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:10.491078    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy" containerID="cri-o://4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2" gracePeriod=30
Nov 02 17:35:10.510582 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:10.510540    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"]
Nov 02 17:35:10.510772 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:10.510752    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx" podUID="27673e46-a4c9-472a-a222-a535807d7cd0" containerName="istio-proxy" containerID="cri-o://b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f" gracePeriod=30
Nov 02 17:35:10.540469 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:10.540431    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"]
Nov 02 17:35:10.540644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:10.540624    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw" podUID="9057858f-a22e-4075-b45b-776a6b468386" containerName="istio-proxy" containerID="cri-o://b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b" gracePeriod=30
Nov 02 17:35:14.194261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:14.194222    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 17:35:15.811031 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.811007    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:35:15.853947 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.853923    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:15.854245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854220    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-kube-api-access-wvpqx\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-kube-api-access-wvpqx\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854295    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-socket\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854333    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-data\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854431    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-istio-token\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-istio-token\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854454    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-certs\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854581 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854490    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-credential-socket\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854581 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854518    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-envoy\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854581 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854544    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc843f16-b45a-43bc-ac74-3314aa932330-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc843f16-b45a-43bc-ac74-3314aa932330-istiod-ca-cert\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854735 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854627    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc843f16-b45a-43bc-ac74-3314aa932330-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc843f16-b45a-43bc-ac74-3314aa932330-istio-podinfo\") pod \"fc843f16-b45a-43bc-ac74-3314aa932330\" (UID: \"fc843f16-b45a-43bc-ac74-3314aa932330\") "
Nov 02 17:35:15.854735 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854646    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-socket" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.854828 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854760    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-data" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.854984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854955    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-credential-socket" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.855038 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.854980    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-certs" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.855038 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.855028    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.855137 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.855049    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.855137 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.855103    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc843f16-b45a-43bc-ac74-3314aa932330-istiod-ca-cert" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:15.856695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.856669    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-kube-api-access-wvpqx" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "kube-api-access-wvpqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.856997 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.856965    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-envoy" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.857089 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.857028    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-istio-token" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.857301 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.857286    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc843f16-b45a-43bc-ac74-3314aa932330-istio-podinfo" pod "fc843f16-b45a-43bc-ac74-3314aa932330" (UID: "fc843f16-b45a-43bc-ac74-3314aa932330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:35:15.858441 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.858425    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:15.955614 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955576    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-data\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.955785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955621    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-credential-socket\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.955785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955665    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-istio-token\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-istio-token\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.955785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955685    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9057858f-a22e-4075-b45b-776a6b468386-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9057858f-a22e-4075-b45b-776a6b468386-istiod-ca-cert\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.955785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955720    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-credential-socket\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.955785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955751    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-kube-api-access-bhlh2\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-kube-api-access-bhlh2\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955801    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-socket\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955867    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-certs\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955887    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-data" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955901    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-envoy\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955946    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-envoy\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.955986    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-istio-token\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-istio-token\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956023    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27673e46-a4c9-472a-a222-a535807d7cd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27673e46-a4c9-472a-a222-a535807d7cd0-istiod-ca-cert\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.956046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956020    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-credential-socket" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956052    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9057858f-a22e-4075-b45b-776a6b468386-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9057858f-a22e-4075-b45b-776a6b468386-istio-podinfo\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956104    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27673e46-a4c9-472a-a222-a535807d7cd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27673e46-a4c9-472a-a222-a535807d7cd0-istio-podinfo\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956171    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-certs\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956197    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-socket" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956206    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9057858f-a22e-4075-b45b-776a6b468386-istiod-ca-cert" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956262    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-data\") pod \"9057858f-a22e-4075-b45b-776a6b468386\" (UID: \"9057858f-a22e-4075-b45b-776a6b468386\") "
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956293    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-socket\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956328    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-credential-socket" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.956476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956334    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-kube-api-access-67mlk\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-kube-api-access-67mlk\") pod \"27673e46-a4c9-472a-a222-a535807d7cd0\" (UID: \"27673e46-a4c9-472a-a222-a535807d7cd0\") "
Nov 02 17:35:15.956953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.956747    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-certs" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957057    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957086    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957103    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc843f16-b45a-43bc-ac74-3314aa932330-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957138    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957154    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957159    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27673e46-a4c9-472a-a222-a535807d7cd0-istiod-ca-cert" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957168    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9057858f-a22e-4075-b45b-776a6b468386-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957181    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc843f16-b45a-43bc-ac74-3314aa932330-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957195    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957203    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-certs" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957209    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvpqx\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-kube-api-access-wvpqx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957224    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957237    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957253    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc843f16-b45a-43bc-ac74-3314aa932330-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957267    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc843f16-b45a-43bc-ac74-3314aa932330-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957376    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-data" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.957570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.957447    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-socket" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.958960 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.958932    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-istio-token" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.959057 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.958972    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-kube-api-access-67mlk" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "kube-api-access-67mlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.959057 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.959037    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-kube-api-access-bhlh2" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "kube-api-access-bhlh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.959166 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.959063    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-istio-token" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.959569 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.959543    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-envoy" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.959569 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.959550    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9057858f-a22e-4075-b45b-776a6b468386-istio-podinfo" pod "9057858f-a22e-4075-b45b-776a6b468386" (UID: "9057858f-a22e-4075-b45b-776a6b468386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:35:15.959931 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.959914    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-envoy" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.960550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:15.960531    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27673e46-a4c9-472a-a222-a535807d7cd0-istio-podinfo" pod "27673e46-a4c9-472a-a222-a535807d7cd0" (UID: "27673e46-a4c9-472a-a222-a535807d7cd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:35:16.058656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058582    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058609    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058621    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67mlk\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-kube-api-access-67mlk\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058631    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27673e46-a4c9-472a-a222-a535807d7cd0-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058639    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhlh2\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-kube-api-access-bhlh2\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058656 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058651    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058942 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058667    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27673e46-a4c9-472a-a222-a535807d7cd0-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058942 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058676    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9057858f-a22e-4075-b45b-776a6b468386-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058942 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058684    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9057858f-a22e-4075-b45b-776a6b468386-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058942 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058692    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27673e46-a4c9-472a-a222-a535807d7cd0-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058942 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058701    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9057858f-a22e-4075-b45b-776a6b468386-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.058942 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.058709    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27673e46-a4c9-472a-a222-a535807d7cd0-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.234885 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.234854    4956 generic.go:356] "Generic (PLEG): container finished" podID="fc843f16-b45a-43bc-ac74-3314aa932330" containerID="4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2" exitCode=0
Nov 02 17:35:16.235058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.234914    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"
Nov 02 17:35:16.235058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.234963    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" event={"ID":"fc843f16-b45a-43bc-ac74-3314aa932330","Type":"ContainerDied","Data":"4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2"}
Nov 02 17:35:16.235058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.235028    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7" event={"ID":"fc843f16-b45a-43bc-ac74-3314aa932330","Type":"ContainerDied","Data":"aac4d6a8ba90af82da18a5359becc24c333c20edb36a37ada59fb78ec96c25e5"}
Nov 02 17:35:16.235264 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.235059    4956 scope.go:122] "RemoveContainer" containerID="4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2"
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.239591    4956 generic.go:356] "Generic (PLEG): container finished" podID="9057858f-a22e-4075-b45b-776a6b468386" containerID="b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b" exitCode=0
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.239939    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw" event={"ID":"9057858f-a22e-4075-b45b-776a6b468386","Type":"ContainerDied","Data":"b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b"}
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.240026    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw" event={"ID":"9057858f-a22e-4075-b45b-776a6b468386","Type":"ContainerDied","Data":"665d1ac82f77575a7da8917b5a857019988f99f8e0d782ddce87bdf02fe65c2c"}
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.240144    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.242840    4956 generic.go:356] "Generic (PLEG): container finished" podID="27673e46-a4c9-472a-a222-a535807d7cd0" containerID="b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f" exitCode=0
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.242868    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx" event={"ID":"27673e46-a4c9-472a-a222-a535807d7cd0","Type":"ContainerDied","Data":"b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f"}
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.242914    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx" event={"ID":"27673e46-a4c9-472a-a222-a535807d7cd0","Type":"ContainerDied","Data":"65b820b7d7591802ef21ec9b454e79e4ccee463b8d9125ad6a450f59ef6b8edf"}
Nov 02 17:35:16.244272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.242923    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"
Nov 02 17:35:16.250402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.250381    4956 scope.go:122] "RemoveContainer" containerID="4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2"
Nov 02 17:35:16.250805 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:16.250742    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2\": container with ID starting with 4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2 not found: ID does not exist" containerID="4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2"
Nov 02 17:35:16.250934 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.250798    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2"} err="failed to get container status \"4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2\": rpc error: code = NotFound desc = could not find container \"4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2\": container with ID starting with 4172cf4468321b6b3879a986275af65da5b3b38be52cead71006b1ea30cc61b2 not found: ID does not exist"
Nov 02 17:35:16.250934 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.250824    4956 scope.go:122] "RemoveContainer" containerID="b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b"
Nov 02 17:35:16.259995 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.259975    4956 scope.go:122] "RemoveContainer" containerID="b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b"
Nov 02 17:35:16.260292 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:16.260273    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b\": container with ID starting with b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b not found: ID does not exist" containerID="b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b"
Nov 02 17:35:16.260366 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.260301    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b"} err="failed to get container status \"b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b\": rpc error: code = NotFound desc = could not find container \"b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b\": container with ID starting with b4e3410576bdb8f740aa86ad22c56c052aee9741a447130b61955b347bb0e41b not found: ID does not exist"
Nov 02 17:35:16.260366 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.260318    4956 scope.go:122] "RemoveContainer" containerID="b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f"
Nov 02 17:35:16.270317 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.270298    4956 scope.go:122] "RemoveContainer" containerID="b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f"
Nov 02 17:35:16.270635 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:16.270613    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f\": container with ID starting with b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f not found: ID does not exist" containerID="b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f"
Nov 02 17:35:16.270705 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.270644    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f"} err="failed to get container status \"b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f\": rpc error: code = NotFound desc = could not find container \"b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f\": container with ID starting with b3237a92cc556e9cd60f0c504a98a041d21826e40585f2f110253e183cfd684f not found: ID does not exist"
Nov 02 17:35:16.390497 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.390419    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"]
Nov 02 17:35:16.396329 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.396298    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xf7sw"]
Nov 02 17:35:16.503275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.503242    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"]
Nov 02 17:35:16.508444 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.508418    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-jkgfx"]
Nov 02 17:35:16.621160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.621098    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"]
Nov 02 17:35:16.625013 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.624985    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-k4vg7"]
Nov 02 17:35:16.874285 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874245    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"]
Nov 02 17:35:16.874692 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874679    4956 cpu_manager.go:414] "containerMap: removing container" podUID="27673e46-a4c9-472a-a222-a535807d7cd0" containerName="istio-proxy"
Nov 02 17:35:16.874736 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874693    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27673e46-a4c9-472a-a222-a535807d7cd0" containerName="istio-proxy"
Nov 02 17:35:16.874736 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874708    4956 cpu_manager.go:414] "containerMap: removing container" podUID="9057858f-a22e-4075-b45b-776a6b468386" containerName="istio-proxy"
Nov 02 17:35:16.874736 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874713    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9057858f-a22e-4075-b45b-776a6b468386" containerName="istio-proxy"
Nov 02 17:35:16.874736 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874721    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy"
Nov 02 17:35:16.874736 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874726    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy"
Nov 02 17:35:16.874874 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874809    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="27673e46-a4c9-472a-a222-a535807d7cd0" containerName="istio-proxy"
Nov 02 17:35:16.874874 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874819    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="9057858f-a22e-4075-b45b-776a6b468386" containerName="istio-proxy"
Nov 02 17:35:16.874874 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.874824    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" containerName="istio-proxy"
Nov 02 17:35:16.879739 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.879715    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:16.934980 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.934949    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ceb1ae\""
Nov 02 17:35:16.935388 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.935369    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ceb1ae-dockercfg-f8jtx\""
Nov 02 17:35:16.935862 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.935835    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"]
Nov 02 17:35:16.967693 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.967640    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/437e79f2-76bd-4221-a2f8-8bd890420692-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:16.967857 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:16.967793    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvggj\" (UniqueName: \"kubernetes.io/projected/437e79f2-76bd-4221-a2f8-8bd890420692-kube-api-access-xvggj\") pod \"payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:17.047041 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.047002    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"]
Nov 02 17:35:17.050711 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.050694    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.068689 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.068660    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvggj\" (UniqueName: \"kubernetes.io/projected/437e79f2-76bd-4221-a2f8-8bd890420692-kube-api-access-xvggj\") pod \"payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:17.068829 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.068748    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/437e79f2-76bd-4221-a2f8-8bd890420692-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:17.069408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.069387    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/437e79f2-76bd-4221-a2f8-8bd890420692-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:17.113528 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.113491    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"]
Nov 02 17:35:17.127219 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.127154    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvggj\" (UniqueName: \"kubernetes.io/projected/437e79f2-76bd-4221-a2f8-8bd890420692-kube-api-access-xvggj\") pod \"payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:17.169394 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.169331    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msv4x\" (UniqueName: \"kubernetes.io/projected/21f4460b-b73d-4949-827f-c59c14e5348d-kube-api-access-msv4x\") pod \"payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.169610 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.169474    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f4460b-b73d-4949-827f-c59c14e5348d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.192784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.192750    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:17.283470 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.283422    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f4460b-b73d-4949-827f-c59c14e5348d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.283636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.283618    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msv4x\" (UniqueName: \"kubernetes.io/projected/21f4460b-b73d-4949-827f-c59c14e5348d-kube-api-access-msv4x\") pod \"payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.285537 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.284854    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f4460b-b73d-4949-827f-c59c14e5348d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.343079 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.342788    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msv4x\" (UniqueName: \"kubernetes.io/projected/21f4460b-b73d-4949-827f-c59c14e5348d-kube-api-access-msv4x\") pod \"payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.365537 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.365500    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:17.591876 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.591853    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"]
Nov 02 17:35:17.593762 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:17.593734    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437e79f2_76bd_4221_a2f8_8bd890420692.slice/crio-a267558825ebcd679621af31c8ba543e4f61af277ae07cec3d55627cc7329eeb WatchSource:0}: Error finding container a267558825ebcd679621af31c8ba543e4f61af277ae07cec3d55627cc7329eeb: Status 404 returned error can't find the container with id a267558825ebcd679621af31c8ba543e4f61af277ae07cec3d55627cc7329eeb
Nov 02 17:35:17.756064 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.755918    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"]
Nov 02 17:35:17.759069 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:17.759038    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f4460b_b73d_4949_827f_c59c14e5348d.slice/crio-a254af3c15f77167d4b8f956c36ad9c482801248a653a6852b4d18c36c6c5224 WatchSource:0}: Error finding container a254af3c15f77167d4b8f956c36ad9c482801248a653a6852b4d18c36c6c5224: Status 404 returned error can't find the container with id a254af3c15f77167d4b8f956c36ad9c482801248a653a6852b4d18c36c6c5224
Nov 02 17:35:17.866032 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.866003    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27673e46-a4c9-472a-a222-a535807d7cd0" path="/var/lib/kubelet/pods/27673e46-a4c9-472a-a222-a535807d7cd0/volumes"
Nov 02 17:35:17.866518 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.866499    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9057858f-a22e-4075-b45b-776a6b468386" path="/var/lib/kubelet/pods/9057858f-a22e-4075-b45b-776a6b468386/volumes"
Nov 02 17:35:17.866902 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:17.866889    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc843f16-b45a-43bc-ac74-3314aa932330" path="/var/lib/kubelet/pods/fc843f16-b45a-43bc-ac74-3314aa932330/volumes"
Nov 02 17:35:18.188009 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.187976    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 17:35:18.188522 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.188045    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 17:35:18.260323 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.260281    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" event={"ID":"21f4460b-b73d-4949-827f-c59c14e5348d","Type":"ContainerStarted","Data":"71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab"}
Nov 02 17:35:18.260635 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.260612    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:18.260731 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.260653    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" event={"ID":"21f4460b-b73d-4949-827f-c59c14e5348d","Type":"ContainerStarted","Data":"a254af3c15f77167d4b8f956c36ad9c482801248a653a6852b4d18c36c6c5224"}
Nov 02 17:35:18.262005 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.261983    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" event={"ID":"437e79f2-76bd-4221-a2f8-8bd890420692","Type":"ContainerStarted","Data":"a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f"}
Nov 02 17:35:18.262137 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.262019    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" event={"ID":"437e79f2-76bd-4221-a2f8-8bd890420692","Type":"ContainerStarted","Data":"a267558825ebcd679621af31c8ba543e4f61af277ae07cec3d55627cc7329eeb"}
Nov 02 17:35:18.262243 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.262226    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:18.552255 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:18.552197    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" podStartSLOduration=2.552180909 podStartE2EDuration="2.552180909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:18.550417206 +0000 UTC m=+748.807447822" watchObservedRunningTime="2026-11-02 17:35:18.552180909 +0000 UTC m=+748.809211532"
Nov 02 17:35:22.839813 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:22.839757    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" podStartSLOduration=6.8397376229999995 podStartE2EDuration="6.839737623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:18.668405797 +0000 UTC m=+748.925436408" watchObservedRunningTime="2026-11-02 17:35:22.839737623 +0000 UTC m=+753.096768242"
Nov 02 17:35:22.840717 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:22.840698    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"]
Nov 02 17:35:22.840913 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:22.840889    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy" containerID="cri-o://3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7" gracePeriod=30
Nov 02 17:35:26.305283 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:26.305153    4956 generic.go:356] "Generic (PLEG): container finished" podID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerID="bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0" exitCode=0
Nov 02 17:35:26.305949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:26.305848    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" event={"ID":"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3","Type":"ContainerDied","Data":"bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0"}
Nov 02 17:35:26.307756 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:26.307739    4956 scope.go:122] "RemoveContainer" containerID="bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0"
Nov 02 17:35:26.309082 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:26.309054    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:26.309165 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:26.309154    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:26.309217 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:26.309189    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:27.312547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.312513    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" event={"ID":"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3","Type":"ContainerStarted","Data":"b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96"}
Nov 02 17:35:27.416269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.416232    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"]
Nov 02 17:35:27.420135 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.420101    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.454029 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.454002    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"]
Nov 02 17:35:27.471826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.471794    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ceb1ae-serving-cert\""
Nov 02 17:35:27.519844 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.519808    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-metrics-certs\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.520048 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.519893    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62hrp\" (UniqueName: \"kubernetes.io/projected/8ce42215-7e35-4588-9083-54c2c9611934-kube-api-access-62hrp\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.520048 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.519984    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.520216 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.520049    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce42215-7e35-4588-9083-54c2c9611934-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.621719 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.621608    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce42215-7e35-4588-9083-54c2c9611934-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.621719 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.621695    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-metrics-certs\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.621966 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.621800    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62hrp\" (UniqueName: \"kubernetes.io/projected/8ce42215-7e35-4588-9083-54c2c9611934-kube-api-access-62hrp\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.621966 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.621837    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.622082 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:27.621971    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-serving-cert: secret "maas-api-e2e-worker-w0-ceb1ae-serving-cert" not found
Nov 02 17:35:27.622082 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:27.622040    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls podName:8ce42215-7e35-4588-9083-54c2c9611934 nodeName:}" failed. No retries permitted until 2026-11-02 17:35:28.122019835 +0000 UTC m=+758.379050447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls") pod "maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r" (UID: "8ce42215-7e35-4588-9083-54c2c9611934") : secret "maas-api-e2e-worker-w0-ceb1ae-serving-cert" not found
Nov 02 17:35:27.622354 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.622332    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce42215-7e35-4588-9083-54c2c9611934-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.624333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.624301    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-metrics-certs\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:27.688384 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:27.688348    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62hrp\" (UniqueName: \"kubernetes.io/projected/8ce42215-7e35-4588-9083-54c2c9611934-kube-api-access-62hrp\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:28.007283 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.007236    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:35:28.007579 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.007556    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 17:35:28.007680 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.007613    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 17:35:28.156301 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.156269    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:35:28.158456 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.158431    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:28.161009 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.160986    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:28.259104 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.258998    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-socket\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259107    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-istio-token\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-istio-token\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259179    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-certs\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259241    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-envoy\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259301    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-socket" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:28.259460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259318    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-kube-api-access-vml5v\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-kube-api-access-vml5v\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259350    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-credential-socket\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259392    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/787ce83b-686e-453e-90d5-61381a488a68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787ce83b-686e-453e-90d5-61381a488a68-istiod-ca-cert\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259401    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-certs" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:28.259460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259436    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/787ce83b-686e-453e-90d5-61381a488a68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787ce83b-686e-453e-90d5-61381a488a68-istio-podinfo\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259747 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259477    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-data\") pod \"787ce83b-686e-453e-90d5-61381a488a68\" (UID: \"787ce83b-686e-453e-90d5-61381a488a68\") "
Nov 02 17:35:28.259747 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259559    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-credential-socket" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:28.259844 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259785    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.259844 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259802    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.259844 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.259817    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.260313 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.260281    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-data" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:28.260612 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.260592    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787ce83b-686e-453e-90d5-61381a488a68-istiod-ca-cert" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:28.262007 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.261978    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-istio-token" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:28.262439 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.262412    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-kube-api-access-vml5v" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "kube-api-access-vml5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:28.262439 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.262394    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-envoy" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:28.262724 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.262704    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/787ce83b-686e-453e-90d5-61381a488a68-istio-podinfo" pod "787ce83b-686e-453e-90d5-61381a488a68" (UID: "787ce83b-686e-453e-90d5-61381a488a68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:35:28.318985 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.318954    4956 generic.go:356] "Generic (PLEG): container finished" podID="787ce83b-686e-453e-90d5-61381a488a68" containerID="3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7" exitCode=0
Nov 02 17:35:28.319451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.319005    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"
Nov 02 17:35:28.319451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.319043    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" event={"ID":"787ce83b-686e-453e-90d5-61381a488a68","Type":"ContainerDied","Data":"3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7"}
Nov 02 17:35:28.319451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.319103    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w" event={"ID":"787ce83b-686e-453e-90d5-61381a488a68","Type":"ContainerDied","Data":"70fb88ccaa9b9435cf2aa97a29c2b3466ece90605f2c04fa1b09a03b96ed0194"}
Nov 02 17:35:28.319451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.319156    4956 scope.go:122] "RemoveContainer" containerID="3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7"
Nov 02 17:35:28.353387 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.353097    4956 scope.go:122] "RemoveContainer" containerID="09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275"
Nov 02 17:35:28.359166 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.359104    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:28.360523 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.360497    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.360644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.360526    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vml5v\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-kube-api-access-vml5v\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.360644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.360537    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787ce83b-686e-453e-90d5-61381a488a68-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.360644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.360546    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787ce83b-686e-453e-90d5-61381a488a68-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.360644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.360561    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787ce83b-686e-453e-90d5-61381a488a68-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.360644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.360574    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787ce83b-686e-453e-90d5-61381a488a68-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:28.363060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.363039    4956 scope.go:122] "RemoveContainer" containerID="3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7"
Nov 02 17:35:28.363425 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:28.363404    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7\": container with ID starting with 3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7 not found: ID does not exist" containerID="3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7"
Nov 02 17:35:28.363503 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.363439    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7"} err="failed to get container status \"3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7\": rpc error: code = NotFound desc = could not find container \"3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7\": container with ID starting with 3b1ae6a550117d1021906dc81de647c01f68e2f63be47964644f7d9e6667a4e7 not found: ID does not exist"
Nov 02 17:35:28.363503 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.363462    4956 scope.go:122] "RemoveContainer" containerID="09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275"
Nov 02 17:35:28.363697 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:28.363678    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275\": container with ID starting with 09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275 not found: ID does not exist" containerID="09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275"
Nov 02 17:35:28.363793 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.363698    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275"} err="failed to get container status \"09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275\": rpc error: code = NotFound desc = could not find container \"09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275\": container with ID starting with 09e56b97b0c143acc4688a33946473a825cdcd90777e4f21f7d9c1c067a9b275 not found: ID does not exist"
Nov 02 17:35:28.485159 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.485110    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"]
Nov 02 17:35:28.492538 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.492494    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-135f915e-openshift-default-5957794c65-gpv4w"]
Nov 02 17:35:28.747148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:28.740013    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"]
Nov 02 17:35:28.752375 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:28.752336    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce42215_7e35_4588_9083_54c2c9611934.slice/crio-aa865d3e2bed90381cfb1d4d5833cdd7d862d99dc161252fc8e3c3d0e2bb0b45 WatchSource:0}: Error finding container aa865d3e2bed90381cfb1d4d5833cdd7d862d99dc161252fc8e3c3d0e2bb0b45: Status 404 returned error can't find the container with id aa865d3e2bed90381cfb1d4d5833cdd7d862d99dc161252fc8e3c3d0e2bb0b45
Nov 02 17:35:29.007860 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:29.007818    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 17:35:29.008039 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:29.007892    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 17:35:29.075188 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:29.075156    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:35:29.269604 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:29.269523    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 17:35:29.269758 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:29.269743    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 17:35:29.345147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:29.325382    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r" event={"ID":"8ce42215-7e35-4588-9083-54c2c9611934","Type":"ContainerStarted","Data":"aa865d3e2bed90381cfb1d4d5833cdd7d862d99dc161252fc8e3c3d0e2bb0b45"}
Nov 02 17:35:29.863460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:29.863380    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787ce83b-686e-453e-90d5-61381a488a68" path="/var/lib/kubelet/pods/787ce83b-686e-453e-90d5-61381a488a68/volumes"
Nov 02 17:35:30.008024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:30.007989    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 17:35:30.008248 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:30.008058    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 17:35:30.360930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:30.360894    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r" event={"ID":"8ce42215-7e35-4588-9083-54c2c9611934","Type":"ContainerStarted","Data":"84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec"}
Nov 02 17:35:30.361387 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:30.361055    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:30.486022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:30.485960    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r" podStartSLOduration=2.629980739 podStartE2EDuration="3.485942496s" totalImagesPullingTime="855.961757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:30.483441795 +0000 UTC m=+760.740472416" watchObservedRunningTime="2026-11-02 17:35:30.485942496 +0000 UTC m=+760.742973115"
Nov 02 17:35:31.008294 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:31.008257    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 17:35:31.008480 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:31.008315    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 17:35:32.007429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:32.007391    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 17:35:32.007875 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:32.007448    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 17:35:33.011496 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:33.011463    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:35:33.012006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:33.011871    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:35:33.012602 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:33.012582    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 17:35:36.372423 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.372393    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 17:35:36.768421 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.768386    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"]
Nov 02 17:35:36.768829 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.768817    4956 cpu_manager.go:414] "containerMap: removing container" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy"
Nov 02 17:35:36.768872 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.768831    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy"
Nov 02 17:35:36.768872 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.768847    4956 cpu_manager.go:414] "containerMap: removing container" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy"
Nov 02 17:35:36.768872 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.768853    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy"
Nov 02 17:35:36.768968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.768918    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy"
Nov 02 17:35:36.768968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.768928    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="787ce83b-686e-453e-90d5-61381a488a68" containerName="istio-proxy"
Nov 02 17:35:36.772286 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.772267    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.826872 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.826840    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"kube-root-ca.crt\""
Nov 02 17:35:36.827072 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.826875    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"openshift-service-ca.crt\""
Nov 02 17:35:36.845534 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.845504    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-model-cache\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.845689 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.845545    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-home\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.845689 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.845571    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tmp-dir\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.845689 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.845678    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-dshm\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.845812 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.845698    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rgx\" (UniqueName: \"kubernetes.io/projected/b6de910d-3a7e-47f8-957f-3d2735c9bf27-kube-api-access-c2rgx\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.845846 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.845809    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tls-certs\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.876224 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.876190    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"default-dockercfg-2jpz8\""
Nov 02 17:35:36.876365 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.876303    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"simulator-w2-a099c3-kserve-self-signed-certs\""
Nov 02 17:35:36.878264 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.878232    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"]
Nov 02 17:35:36.946491 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.946453    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tls-certs\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.946694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.946509    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-model-cache\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.946694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.946536    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-home\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.946694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.946560    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tmp-dir\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.946694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.946583    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-dshm\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.946694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.946602    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rgx\" (UniqueName: \"kubernetes.io/projected/b6de910d-3a7e-47f8-957f-3d2735c9bf27-kube-api-access-c2rgx\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.947005 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.946979    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-home\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.947148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.947029    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-model-cache\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.947148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.947097    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tmp-dir\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.949101 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.949083    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-dshm\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:36.949255 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:36.949241    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tls-certs\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:37.003997 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:37.003969    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rgx\" (UniqueName: \"kubernetes.io/projected/b6de910d-3a7e-47f8-957f-3d2735c9bf27-kube-api-access-c2rgx\") pod \"simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") " pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:37.084109 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:37.084029    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:37.458561 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:37.458534    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"]
Nov 02 17:35:37.460539 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:37.460511    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6de910d_3a7e_47f8_957f_3d2735c9bf27.slice/crio-5ee43ff21dc1d326ea57177bc61948d25d2aebecf97fcdcf9d4adccbc19fdb65 WatchSource:0}: Error finding container 5ee43ff21dc1d326ea57177bc61948d25d2aebecf97fcdcf9d4adccbc19fdb65: Status 404 returned error can't find the container with id 5ee43ff21dc1d326ea57177bc61948d25d2aebecf97fcdcf9d4adccbc19fdb65
Nov 02 17:35:38.324481 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.324445    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"]
Nov 02 17:35:38.328057 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.328040    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.360024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.359992    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.360212 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.360036    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffg4\" (UniqueName: \"kubernetes.io/projected/416af64b-0e3c-4d42-b76c-5465fa83b650-kube-api-access-6ffg4\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.360260 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.360217    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.360306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.360269    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416af64b-0e3c-4d42-b76c-5465fa83b650-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.379858 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.379819    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4ad655-serving-cert\""
Nov 02 17:35:38.403189 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.403149    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" event={"ID":"b6de910d-3a7e-47f8-957f-3d2735c9bf27","Type":"ContainerStarted","Data":"69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e"}
Nov 02 17:35:38.403189 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.403196    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" event={"ID":"b6de910d-3a7e-47f8-957f-3d2735c9bf27","Type":"ContainerStarted","Data":"5ee43ff21dc1d326ea57177bc61948d25d2aebecf97fcdcf9d4adccbc19fdb65"}
Nov 02 17:35:38.431520 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.431487    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"]
Nov 02 17:35:38.461022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.460982    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416af64b-0e3c-4d42-b76c-5465fa83b650-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.461522 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.461057    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.461522 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.461093    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffg4\" (UniqueName: \"kubernetes.io/projected/416af64b-0e3c-4d42-b76c-5465fa83b650-kube-api-access-6ffg4\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.461522 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.461260    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.461522 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:38.461367    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-serving-cert: secret "maas-api-e2e-worker-w4-4ad655-serving-cert" not found
Nov 02 17:35:38.461522 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:38.461426    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls podName:416af64b-0e3c-4d42-b76c-5465fa83b650 nodeName:}" failed. No retries permitted until 2026-11-02 17:35:38.96140665 +0000 UTC m=+769.218437265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls") pod "maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv" (UID: "416af64b-0e3c-4d42-b76c-5465fa83b650") : secret "maas-api-e2e-worker-w4-4ad655-serving-cert" not found
Nov 02 17:35:38.461759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.461740    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416af64b-0e3c-4d42-b76c-5465fa83b650-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.463984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.463957    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.530836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.530805    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffg4\" (UniqueName: \"kubernetes.io/projected/416af64b-0e3c-4d42-b76c-5465fa83b650-kube-api-access-6ffg4\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.606354 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.606221    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" podStartSLOduration=2.606205881 podStartE2EDuration="2.606205881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:38.604030778 +0000 UTC m=+768.861061397" watchObservedRunningTime="2026-11-02 17:35:38.606205881 +0000 UTC m=+768.863236500"
Nov 02 17:35:38.967489 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.967450    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:38.969983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:38.969955    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:39.084433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:39.084396    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:39.242974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:39.242891    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:39.665946 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:39.665917    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"]
Nov 02 17:35:39.667361 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:35:39.667329    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416af64b_0e3c_4d42_b76c_5465fa83b650.slice/crio-a0488939695098b680b2f77550d60d87efd03a5e920bd9c89f0a4d50e4e2c835 WatchSource:0}: Error finding container a0488939695098b680b2f77550d60d87efd03a5e920bd9c89f0a4d50e4e2c835: Status 404 returned error can't find the container with id a0488939695098b680b2f77550d60d87efd03a5e920bd9c89f0a4d50e4e2c835
Nov 02 17:35:40.414816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:40.414781    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv" event={"ID":"416af64b-0e3c-4d42-b76c-5465fa83b650","Type":"ContainerStarted","Data":"a0488939695098b680b2f77550d60d87efd03a5e920bd9c89f0a4d50e4e2c835"}
Nov 02 17:35:41.421028 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:41.420986    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv" event={"ID":"416af64b-0e3c-4d42-b76c-5465fa83b650","Type":"ContainerStarted","Data":"e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f"}
Nov 02 17:35:41.421474 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:41.421137    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:41.539244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:41.539190    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv" podStartSLOduration=2.70879353 podStartE2EDuration="3.53917458s" totalImagesPullingTime="830.38105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:41.537811057 +0000 UTC m=+771.794841676" watchObservedRunningTime="2026-11-02 17:35:41.53917458 +0000 UTC m=+771.796205201"
Nov 02 17:35:47.084953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:47.084913    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:47.097522 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:47.097489    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:47.432227 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:47.432149    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 17:35:47.464010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:35:47.463979    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:35:59.052079 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:35:59.052045    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:36:24.071194 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.071160    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"]
Nov 02 17:36:24.074915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.074897    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.107982 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.107952    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-home\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.108184 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.107999    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-dshm\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.108184 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.108033    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqql6\" (UniqueName: \"kubernetes.io/projected/b6389d6f-249f-4998-8760-30c38316ba91-kube-api-access-bqql6\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.108306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.108189    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-tmp-dir\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.108306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.108234    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-model-cache\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.108417 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.108388    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6389d6f-249f-4998-8760-30c38316ba91-tls-certs\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.178395 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.178355    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"premium-w2-a099c3-kserve-self-signed-certs\""
Nov 02 17:36:24.180450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.180426    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"]
Nov 02 17:36:24.209525 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.209490    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-tmp-dir\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.209525 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.209529    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-model-cache\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.209761 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.209595    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6389d6f-249f-4998-8760-30c38316ba91-tls-certs\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.209761 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.209646    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-home\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.209761 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.209692    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-dshm\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.209761 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.209735    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqql6\" (UniqueName: \"kubernetes.io/projected/b6389d6f-249f-4998-8760-30c38316ba91-kube-api-access-bqql6\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.209987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.209941    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-tmp-dir\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.210168 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.210145    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-model-cache\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.210428 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.210405    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-home\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.212543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.212508    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-dshm\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.213824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.213793    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6389d6f-249f-4998-8760-30c38316ba91-tls-certs\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.267692 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.267663    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqql6\" (UniqueName: \"kubernetes.io/projected/b6389d6f-249f-4998-8760-30c38316ba91-kube-api-access-bqql6\") pod \"premium-w2-a099c3-kserve-7779d88cf5-rbj2w\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") " pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.386314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.386171    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:24.762757 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:24.762732    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"]
Nov 02 17:36:24.765270 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:36:24.765238    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6389d6f_249f_4998_8760_30c38316ba91.slice/crio-ef4292e3b104935c8f902a45dc332e94018c5c411c659cc83f89c0623468f7dd WatchSource:0}: Error finding container ef4292e3b104935c8f902a45dc332e94018c5c411c659cc83f89c0623468f7dd: Status 404 returned error can't find the container with id ef4292e3b104935c8f902a45dc332e94018c5c411c659cc83f89c0623468f7dd
Nov 02 17:36:25.654837 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:25.654795    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" event={"ID":"b6389d6f-249f-4998-8760-30c38316ba91","Type":"ContainerStarted","Data":"42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729"}
Nov 02 17:36:25.654837 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:25.654838    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" event={"ID":"b6389d6f-249f-4998-8760-30c38316ba91","Type":"ContainerStarted","Data":"ef4292e3b104935c8f902a45dc332e94018c5c411c659cc83f89c0623468f7dd"}
Nov 02 17:36:25.771472 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:25.771418    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" podStartSLOduration=1.771401623 podStartE2EDuration="1.771401623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:25.769519506 +0000 UTC m=+816.026550139" watchObservedRunningTime="2026-11-02 17:36:25.771401623 +0000 UTC m=+816.028432309"
Nov 02 17:36:26.386433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:26.386386    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:29.052913 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:36:29.052875    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:36:29.811388 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.811360    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"]
Nov 02 17:36:29.816272 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.816254    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.866764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.866732    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-dshm\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.866764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.866769    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-home\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.866963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.866800    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj49n\" (UniqueName: \"kubernetes.io/projected/454bbc03-2858-4c60-a7ce-5e3b7079cec1-kube-api-access-hj49n\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.866963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.866941    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-model-cache\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.867036 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.867005    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tls-certs\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.867084 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.867056    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tmp-dir\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.869363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.869335    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"openshift-service-ca.crt\""
Nov 02 17:36:29.871287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.871260    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"default-dockercfg-kdg69\""
Nov 02 17:36:29.871425 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.871346    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"kube-root-ca.crt\""
Nov 02 17:36:29.871425 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.871413    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"simulator-w0-ceb1ae-kserve-self-signed-certs\""
Nov 02 17:36:29.872038 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.872019    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"]
Nov 02 17:36:29.968097 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968053    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj49n\" (UniqueName: \"kubernetes.io/projected/454bbc03-2858-4c60-a7ce-5e3b7079cec1-kube-api-access-hj49n\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968097 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968099    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-model-cache\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968207    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tls-certs\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968264    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tmp-dir\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968417    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-dshm\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968611 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968449    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-home\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968669 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968619    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-model-cache\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968746 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968721    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tmp-dir\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.968810 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.968783    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-home\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.971269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.971245    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tls-certs\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:29.971269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:29.971257    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-dshm\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:30.025481 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:30.025443    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj49n\" (UniqueName: \"kubernetes.io/projected/454bbc03-2858-4c60-a7ce-5e3b7079cec1-kube-api-access-hj49n\") pod \"simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") " pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:30.128576 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:30.128484    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:30.499262 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:30.499221    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"]
Nov 02 17:36:30.503153 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:36:30.503111    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod454bbc03_2858_4c60_a7ce_5e3b7079cec1.slice/crio-516ac45813368ccf1f8d75096586a07a82b12ebd6fb1eb13c4a959071b424685 WatchSource:0}: Error finding container 516ac45813368ccf1f8d75096586a07a82b12ebd6fb1eb13c4a959071b424685: Status 404 returned error can't find the container with id 516ac45813368ccf1f8d75096586a07a82b12ebd6fb1eb13c4a959071b424685
Nov 02 17:36:30.681101 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:30.681055    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" event={"ID":"454bbc03-2858-4c60-a7ce-5e3b7079cec1","Type":"ContainerStarted","Data":"d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9"}
Nov 02 17:36:30.681273 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:30.681110    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" event={"ID":"454bbc03-2858-4c60-a7ce-5e3b7079cec1","Type":"ContainerStarted","Data":"516ac45813368ccf1f8d75096586a07a82b12ebd6fb1eb13c4a959071b424685"}
Nov 02 17:36:30.795472 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:30.795360    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" podStartSLOduration=1.795344424 podStartE2EDuration="1.795344424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:30.794368567 +0000 UTC m=+821.051399187" watchObservedRunningTime="2026-11-02 17:36:30.795344424 +0000 UTC m=+821.052375122"
Nov 02 17:36:31.128985 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:31.128888    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:34.386371 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:34.386327    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:34.399271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:34.399242    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:34.712284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:34.712253    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:36:40.128772 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:40.128744    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:40.141796 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:40.141768    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:40.745220 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:36:40.745193    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:36:59.053035 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:36:59.052990    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:37:06.125616 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.125530    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"]
Nov 02 17:37:06.129918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.129896    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.229694 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.229665    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"openshift-service-ca.crt\""
Nov 02 17:37:06.232017 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.231981    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"kube-root-ca.crt\""
Nov 02 17:37:06.232187 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.232055    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"default-dockercfg-gsl42\""
Nov 02 17:37:06.232459 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.232430    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"simulator-w1-dc5689-kserve-self-signed-certs\""
Nov 02 17:37:06.237684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.237659    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"]
Nov 02 17:37:06.240412 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.240384    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tmp-dir\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.240508 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.240445    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tls-certs\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.240549 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.240526    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-model-cache\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.240586 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.240561    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-home\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.240669 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.240648    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbgc\" (UniqueName: \"kubernetes.io/projected/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-kube-api-access-qkbgc\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.240746 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.240733    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-dshm\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342262 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342214    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tls-certs\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342449 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342283    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-model-cache\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342449 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342320    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-home\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342449 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342373    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbgc\" (UniqueName: \"kubernetes.io/projected/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-kube-api-access-qkbgc\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342459    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-dshm\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342496    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tmp-dir\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342778    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-home\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342810    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-model-cache\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.342994 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.342838    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tmp-dir\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.344874 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.344842    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-dshm\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.345043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.345025    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tls-certs\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.399475 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.399400    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbgc\" (UniqueName: \"kubernetes.io/projected/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-kube-api-access-qkbgc\") pod \"simulator-w1-dc5689-kserve-794866f44c-4xt29\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") " pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.469564 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.469513    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:06.851889 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.851855    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"]
Nov 02 17:37:06.854335 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:37:06.854305    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcdec34f_6eaf_44f6_920e_1deaa0af0c4a.slice/crio-70782368248aef025b3ba3b5da0f4325a17eba8de8f93351f5cdf20fd0c2a9e7 WatchSource:0}: Error finding container 70782368248aef025b3ba3b5da0f4325a17eba8de8f93351f5cdf20fd0c2a9e7: Status 404 returned error can't find the container with id 70782368248aef025b3ba3b5da0f4325a17eba8de8f93351f5cdf20fd0c2a9e7
Nov 02 17:37:06.871176 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:06.871143    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" event={"ID":"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a","Type":"ContainerStarted","Data":"70782368248aef025b3ba3b5da0f4325a17eba8de8f93351f5cdf20fd0c2a9e7"}
Nov 02 17:37:07.877943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:07.877912    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" event={"ID":"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a","Type":"ContainerStarted","Data":"1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d"}
Nov 02 17:37:07.995757 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:07.995705    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" podStartSLOduration=1.99569029 podStartE2EDuration="1.99569029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:07.994069491 +0000 UTC m=+858.251100113" watchObservedRunningTime="2026-11-02 17:37:07.99569029 +0000 UTC m=+858.252720951"
Nov 02 17:37:08.470628 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:08.470598    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:16.470389 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:16.470348    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:16.483167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:16.483141    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:16.934378 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:16.934303    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:37:29.053057 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:37:29.053020    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:37:49.951893 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:49.951860    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:37:49.952544 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:37:49.952105    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:37:59.052457 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:37:59.052418    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:38:11.179789 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.179746    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"]
Nov 02 17:38:11.183803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.183780    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.219573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.219535    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-dshm\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.219573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.219576    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tls-certs\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.219794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.219608    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-model-cache\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.219794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.219736    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-home\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.219794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.219771    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tmp-dir\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.219900 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.219869    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4th\" (UniqueName: \"kubernetes.io/projected/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-kube-api-access-2n4th\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.286982 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.286953    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"premium-w1-dc5689-kserve-self-signed-certs\""
Nov 02 17:38:11.289297 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.289271    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"]
Nov 02 17:38:11.320939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.320904    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-dshm\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.321156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.320946    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tls-certs\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.321156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.320978    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-model-cache\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.321156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.321021    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-home\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.321156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.321049    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tmp-dir\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.321729 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.321167    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4th\" (UniqueName: \"kubernetes.io/projected/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-kube-api-access-2n4th\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.321729 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.321480    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-home\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.322193 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.322169    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-model-cache\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.322346 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.322325    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tmp-dir\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.323793 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.323771    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-dshm\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.323871 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.323846    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tls-certs\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.378871 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.378837    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4th\" (UniqueName: \"kubernetes.io/projected/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-kube-api-access-2n4th\") pod \"premium-w1-dc5689-kserve-76968c6dbb-w9xhg\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") " pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.495964 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.495927    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:11.875674 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:11.875644    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"]
Nov 02 17:38:11.877331 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:38:11.877303    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e101ba8_4c2a_4657_ba2d_2164a2bd890d.slice/crio-8a2df0e8bfd14117581b8267aa08e4e50b73f6cd48e639f0bf9eb7b01297f316 WatchSource:0}: Error finding container 8a2df0e8bfd14117581b8267aa08e4e50b73f6cd48e639f0bf9eb7b01297f316: Status 404 returned error can't find the container with id 8a2df0e8bfd14117581b8267aa08e4e50b73f6cd48e639f0bf9eb7b01297f316
Nov 02 17:38:12.206225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:12.206192    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" event={"ID":"6e101ba8-4c2a-4657-ba2d-2164a2bd890d","Type":"ContainerStarted","Data":"37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb"}
Nov 02 17:38:12.206695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:12.206232    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" event={"ID":"6e101ba8-4c2a-4657-ba2d-2164a2bd890d","Type":"ContainerStarted","Data":"8a2df0e8bfd14117581b8267aa08e4e50b73f6cd48e639f0bf9eb7b01297f316"}
Nov 02 17:38:12.322431 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:12.322373    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" podStartSLOduration=1.322357057 podStartE2EDuration="1.322357057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:12.320401116 +0000 UTC m=+922.577431738" watchObservedRunningTime="2026-11-02 17:38:12.322357057 +0000 UTC m=+922.579387676"
Nov 02 17:38:12.496953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:12.496866    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:21.496341 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:21.496299    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:21.509353 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:21.509321    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:22.272551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:22.272522    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:38:29.052532 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:38:29.052494    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:38:39.056900 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.056820    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"]
Nov 02 17:38:39.060622 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.060606    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.112677 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.112644    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ceb1ae-openshift-default-dockercfg-pmptc\""
Nov 02 17:38:39.164347 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.164315    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"]
Nov 02 17:38:39.230060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230022    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230274 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230088    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230274 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230244    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230324    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230385 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230355    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230425 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230382    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230477 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230457    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230560 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230536    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f0b20-ec54-46da-94b3-99ea645767da-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.230601 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.230581    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfxz\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-kube-api-access-wwfxz\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331412 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331320    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331412 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331372    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331427    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331468    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331501    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331520    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331553    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331582    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f0b20-ec54-46da-94b3-99ea645767da-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.331663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331610    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfxz\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-kube-api-access-wwfxz\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.332029 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331777    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.332029 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331823    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.332029 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.331842    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.332229 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.332062    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.332478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.332459    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f0b20-ec54-46da-94b3-99ea645767da-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.333976 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.333948    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.334195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.334176    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.388602 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.388563    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.388750 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.388694    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfxz\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-kube-api-access-wwfxz\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:39.673815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:39.673728    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:40.101999 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.101849    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"]
Nov 02 17:38:40.104997 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:38:40.104961    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6f0b20_ec54_46da_94b3_99ea645767da.slice/crio-9a7981bc761c2f853c41eab90290bd39b6586f202981ab1c2def5f4df3b41850 WatchSource:0}: Error finding container 9a7981bc761c2f853c41eab90290bd39b6586f202981ab1c2def5f4df3b41850: Status 404 returned error can't find the container with id 9a7981bc761c2f853c41eab90290bd39b6586f202981ab1c2def5f4df3b41850
Nov 02 17:38:40.107341 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.107313    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:40.107402 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.107376    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:40.107438 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.107406    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:40.374858 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.374768    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" event={"ID":"5c6f0b20-ec54-46da-94b3-99ea645767da","Type":"ContainerStarted","Data":"50e3dde6544f39e217acaf6b8237fddb8f74b0ff59eb29443d1986b1a32c6986"}
Nov 02 17:38:40.374858 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.374824    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" event={"ID":"5c6f0b20-ec54-46da-94b3-99ea645767da","Type":"ContainerStarted","Data":"9a7981bc761c2f853c41eab90290bd39b6586f202981ab1c2def5f4df3b41850"}
Nov 02 17:38:40.493030 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.492974    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podStartSLOduration=1.4929579880000001 podStartE2EDuration="1.492957988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:40.491086929 +0000 UTC m=+950.748117549" watchObservedRunningTime="2026-11-02 17:38:40.492957988 +0000 UTC m=+950.749988673"
Nov 02 17:38:40.674780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:40.674693    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:41.675937 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:41.675905    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:38:41.676393 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:41.675956    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 17:38:42.675058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:42.675019    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:38:42.675255 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:42.675079    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 17:38:43.674364 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:43.674326    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:38:43.674768 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:43.674380    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 17:38:44.674315 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:44.674278    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:38:44.674499 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:44.674336    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 17:38:45.674496 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:45.674457    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:38:45.674922 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:45.674525    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 17:38:46.674964 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:46.674904    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:38:46.675488 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:46.675015    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 17:38:47.679481 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:47.679452    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:47.679903 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:47.679726    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:47.680423 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:38:47.680403    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:38:59.056442 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:38:59.056399    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:39:21.996643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:21.996607    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"]
Nov 02 17:39:22.000832 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.000807    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108191 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108152    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108379 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108270    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108379 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108298    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108379 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108354    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac57e2-619f-4ada-abfb-2369d107221c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108382    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108408    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108483    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108517    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4klb\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-kube-api-access-k4klb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.108611 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.108600    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.196293 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.196251    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"]
Nov 02 17:39:22.209833 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.209777    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.209841    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.209881    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac57e2-619f-4ada-abfb-2369d107221c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.209919    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.209949    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.210001    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.210031    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4klb\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-kube-api-access-k4klb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.210110    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.210223    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.210336    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.210531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.210342    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.211032 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.211001    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.211174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.211093    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.211270 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.211248    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac57e2-619f-4ada-abfb-2369d107221c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.213154 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.213113    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.213511 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.213487    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.267180 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.267072    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.267627 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.267608    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4klb\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-kube-api-access-k4klb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-x2fx9\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.313595 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.313562    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:22.739702 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.739677    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"]
Nov 02 17:39:22.740874 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:39:22.740851    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ac57e2_619f_4ada_abfb_2369d107221c.slice/crio-b783fb9623bd5ff82bee792da74fb3ac89e8f823135874280d81966faad4b9e4 WatchSource:0}: Error finding container b783fb9623bd5ff82bee792da74fb3ac89e8f823135874280d81966faad4b9e4: Status 404 returned error can't find the container with id b783fb9623bd5ff82bee792da74fb3ac89e8f823135874280d81966faad4b9e4
Nov 02 17:39:22.743248 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.743220    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:39:22.743330 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.743279    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:39:22.743330 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:22.743310    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:39:23.615662 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:23.615620    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" event={"ID":"a1ac57e2-619f-4ada-abfb-2369d107221c","Type":"ContainerStarted","Data":"08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb"}
Nov 02 17:39:23.616062 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:23.615704    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" event={"ID":"a1ac57e2-619f-4ada-abfb-2369d107221c","Type":"ContainerStarted","Data":"b783fb9623bd5ff82bee792da74fb3ac89e8f823135874280d81966faad4b9e4"}
Nov 02 17:39:23.787170 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:23.787098    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podStartSLOduration=2.787084272 podStartE2EDuration="2.787084272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:23.78532603 +0000 UTC m=+994.042356723" watchObservedRunningTime="2026-11-02 17:39:23.787084272 +0000 UTC m=+994.044114891"
Nov 02 17:39:24.314304 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:24.314260    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:24.315964 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:24.315939    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:24.316082 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:24.315986    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 17:39:25.314759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:25.314721    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:25.315204 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:25.314787    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 17:39:26.314222 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:26.314181    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:26.314457 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:26.314249    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 17:39:27.314365 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:27.314326    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:27.314775 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:27.314382    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 17:39:28.313992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:28.313952    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:28.314211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:28.314017    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 17:39:29.054632 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:39:29.054593    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:39:29.314439 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:29.314346    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:29.314439 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:29.314405    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 17:39:30.314953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:30.314909    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:30.315513 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:30.314974    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 17:39:31.447275 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:31.447238    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:31.448004 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:31.447976    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:31.449941 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:31.449910    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 17:39:37.114795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.114755    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"]
Nov 02 17:39:37.118916 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.118897    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206098 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206065    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206320 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206140    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206320 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206226    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4hl\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-kube-api-access-nr4hl\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206320 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206290    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206349    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206404    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206504 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206437    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827b7174-42fe-4f11-a072-83842c49f869-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206536 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206507    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827b7174-42fe-4f11-a072-83842c49f869-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.206536 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.206530    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.307540 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.307509    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827b7174-42fe-4f11-a072-83842c49f869-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.307540 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.307549    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.307810 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.307651    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.307873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.307802    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.307873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.307843    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4hl\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-kube-api-access-nr4hl\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.307985 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.307879    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308070 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308048    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308077    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308171 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308157    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308183    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827b7174-42fe-4f11-a072-83842c49f869-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308080    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308296 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308176    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308437 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308359    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.308515 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.308492    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827b7174-42fe-4f11-a072-83842c49f869-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.310638 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.310614    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.310743 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.310726    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827b7174-42fe-4f11-a072-83842c49f869-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.323549 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.323519    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"]
Nov 02 17:39:37.367923 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.367850    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.368306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.368271    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4hl\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-kube-api-access-nr4hl\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4g6dg\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.432435 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.432405    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:37.865333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.865294    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"]
Nov 02 17:39:37.899934 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:39:37.899885    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827b7174_42fe_4f11_a072_83842c49f869.slice/crio-fb1553f3d47ca74b66ca7d9861c3400fdf5914397f17ace26dea9fdde9e9e63f WatchSource:0}: Error finding container fb1553f3d47ca74b66ca7d9861c3400fdf5914397f17ace26dea9fdde9e9e63f: Status 404 returned error can't find the container with id fb1553f3d47ca74b66ca7d9861c3400fdf5914397f17ace26dea9fdde9e9e63f
Nov 02 17:39:37.902392 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.902362    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:39:37.902467 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.902434    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:39:37.902504 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:37.902485    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:39:38.723763 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:38.723729    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" event={"ID":"827b7174-42fe-4f11-a072-83842c49f869","Type":"ContainerStarted","Data":"4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739"}
Nov 02 17:39:38.724256 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:38.723781    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" event={"ID":"827b7174-42fe-4f11-a072-83842c49f869","Type":"ContainerStarted","Data":"fb1553f3d47ca74b66ca7d9861c3400fdf5914397f17ace26dea9fdde9e9e63f"}
Nov 02 17:39:38.944146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:38.943464    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podStartSLOduration=1.943439786 podStartE2EDuration="1.943439786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:38.888884683 +0000 UTC m=+1009.145915303" watchObservedRunningTime="2026-11-02 17:39:38.943439786 +0000 UTC m=+1009.200470410"
Nov 02 17:39:39.448147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:39.447533    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:39.450268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:39.450228    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:39.450490 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:39.450297    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 17:39:40.438145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:40.434543    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:40.438145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:40.434605    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 17:39:41.445147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:41.439598    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:41.445147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:41.439668    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 17:39:42.447326 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:42.447228    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:42.448915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:42.448881    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 17:39:43.435146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:43.434648    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:43.435146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:43.434725    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 17:39:44.433146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:44.432831    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:44.433146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:44.432907    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 17:39:45.433991 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:45.433952    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:45.434663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:45.434611    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 17:39:47.432853 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:47.432817    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:39:47.433385 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:47.432884    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:39:48.473307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:48.473262    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:39:48.473792 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:48.473333    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:39:48.967043 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:48.967008    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:48.967603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:48.967579    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:48.968526 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:39:48.968508    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:39:59.057200 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:39:59.057159    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:40:11.212032 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.211995    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"]
Nov 02 17:40:11.216340 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.216315    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.315460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.315426    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-tmp-dir\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.315460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.315462    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d251df15-1ad4-4856-a2e0-939e2cdc083f-tls-certs\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.315698 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.315584    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-model-cache\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.315698 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.315665    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-dshm\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.315769 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.315707    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwnf\" (UniqueName: \"kubernetes.io/projected/d251df15-1ad4-4856-a2e0-939e2cdc083f-kube-api-access-xzwnf\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.315769 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.315739    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-home\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.318767 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.318747    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"kube-root-ca.crt\""
Nov 02 17:40:11.318767 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.318750    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"openshift-service-ca.crt\""
Nov 02 17:40:11.319008 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.318992    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"distinct-w4-4ad655-kserve-self-signed-certs\""
Nov 02 17:40:11.319076 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.319060    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"default-dockercfg-tm5n2\""
Nov 02 17:40:11.321221 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.321200    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"]
Nov 02 17:40:11.416702 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.416671    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-tmp-dir\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.416702 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.416703    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d251df15-1ad4-4856-a2e0-939e2cdc083f-tls-certs\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.416963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.416752    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-model-cache\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.416963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.416778    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-dshm\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.416963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.416810    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwnf\" (UniqueName: \"kubernetes.io/projected/d251df15-1ad4-4856-a2e0-939e2cdc083f-kube-api-access-xzwnf\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.416963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.416850    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-home\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.417218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.417183    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-tmp-dir\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.417286 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.417253    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-model-cache\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.417286 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.417276    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-home\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.419224 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.419198    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-dshm\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.419535 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.419520    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d251df15-1ad4-4856-a2e0-939e2cdc083f-tls-certs\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.474788 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.474711    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwnf\" (UniqueName: \"kubernetes.io/projected/d251df15-1ad4-4856-a2e0-939e2cdc083f-kube-api-access-xzwnf\") pod \"distinct-w4-4ad655-kserve-7c685b8654-d7w5x\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.528485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.528449    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:11.912822 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.912800    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"]
Nov 02 17:40:11.914709 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:40:11.914684    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd251df15_1ad4_4856_a2e0_939e2cdc083f.slice/crio-ee1ce750f7cad1cbce083d89021e0edce328263f46151072ddc73b492d943e86 WatchSource:0}: Error finding container ee1ce750f7cad1cbce083d89021e0edce328263f46151072ddc73b492d943e86: Status 404 returned error can't find the container with id ee1ce750f7cad1cbce083d89021e0edce328263f46151072ddc73b492d943e86
Nov 02 17:40:11.939306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:11.939275    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" event={"ID":"d251df15-1ad4-4856-a2e0-939e2cdc083f","Type":"ContainerStarted","Data":"ee1ce750f7cad1cbce083d89021e0edce328263f46151072ddc73b492d943e86"}
Nov 02 17:40:12.946190 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:12.946156    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" event={"ID":"d251df15-1ad4-4856-a2e0-939e2cdc083f","Type":"ContainerStarted","Data":"f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f"}
Nov 02 17:40:13.063074 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:13.063019    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" podStartSLOduration=2.063004082 podStartE2EDuration="2.063004082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:13.060947471 +0000 UTC m=+1043.317978092" watchObservedRunningTime="2026-11-02 17:40:13.063004082 +0000 UTC m=+1043.320034702"
Nov 02 17:40:13.528685 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:13.528653    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:21.528882 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:21.528852    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:21.541335 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:21.541307    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:22.003323 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:22.003290    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:40:29.052453 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:40:29.052416    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:40:32.735676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.735635    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"]
Nov 02 17:40:32.740172 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.740152    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.792261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.792234    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2296d33cdbd2\""
Nov 02 17:40:32.841763 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.841716    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 17:40:32.841937 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.841716    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 17:40:32.841937 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.841780    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cvktf\""
Nov 02 17:40:32.841937 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.841727    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 17:40:32.841937 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.841727    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 17:40:32.844316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.844298    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"]
Nov 02 17:40:32.850384 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.850366    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-provider-credentials\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.850467 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.850419    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-routing-overlay-volume\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.850616 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.850591    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-plugins-config-volume\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.850755 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.850738    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddc2\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-kube-api-access-nddc2\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.951582 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.951544    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddc2\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-kube-api-access-nddc2\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.951782 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.951609    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-provider-credentials\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.951782 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.951674    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-routing-overlay-volume\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.951782 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.951764    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-plugins-config-volume\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.952399 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.952364    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-routing-overlay-volume\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.952514 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.952443    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-plugins-config-volume\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:32.954146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:32.954109    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-provider-credentials\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:33.007830 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:33.007760    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddc2\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-kube-api-access-nddc2\") pod \"payload-processing-external-model-5d5b9f745f-6qz4w\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:33.054841 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:33.054810    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:33.425727 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:33.425699    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"]
Nov 02 17:40:33.427345 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:40:33.427318    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf50acf_3602_4f70_bdd3_2d476313c6af.slice/crio-0377b2998129561675e41589f82733dfc162adcf42d90b86397723569a604051 WatchSource:0}: Error finding container 0377b2998129561675e41589f82733dfc162adcf42d90b86397723569a604051: Status 404 returned error can't find the container with id 0377b2998129561675e41589f82733dfc162adcf42d90b86397723569a604051
Nov 02 17:40:34.061941 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:34.061906    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w" event={"ID":"3bf50acf-3602-4f70-bdd3-2d476313c6af","Type":"ContainerStarted","Data":"5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477"}
Nov 02 17:40:34.062473 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:34.061962    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:34.062473 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:34.061982    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w" event={"ID":"3bf50acf-3602-4f70-bdd3-2d476313c6af","Type":"ContainerStarted","Data":"0377b2998129561675e41589f82733dfc162adcf42d90b86397723569a604051"}
Nov 02 17:40:34.177200 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:34.177152    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w" podStartSLOduration=2.177138536 podStartE2EDuration="2.177138536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:34.174483192 +0000 UTC m=+1064.431513812" watchObservedRunningTime="2026-11-02 17:40:34.177138536 +0000 UTC m=+1064.434169148"
Nov 02 17:40:41.085974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:41.085937    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"]
Nov 02 17:40:41.089228 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:41.089193    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w" podUID="3bf50acf-3602-4f70-bdd3-2d476313c6af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:40:44.533657 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.533615    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"]
Nov 02 17:40:44.537784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.537761    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.557977    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558015    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558105    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558153    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558177    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558196    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbdfc53-0624-421c-82e0-6b083865f6fd-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558215    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsh7g\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-kube-api-access-jsh7g\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558475 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558403    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.558475 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.558441    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.589500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.589467    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ad655-openshift-default-dockercfg-jbjjg\""
Nov 02 17:40:44.642664 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.642628    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"]
Nov 02 17:40:44.659172 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659109    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659373 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659214    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659373 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659311    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659373 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659340    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659639 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659425    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659639 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659458    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659639 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659490    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659639 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659517    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbdfc53-0624-421c-82e0-6b083865f6fd-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659639 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659542    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsh7g\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-kube-api-access-jsh7g\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659639 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659599    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659941 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659686    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659941 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659761    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.659941 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.659918    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.660373 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.660353    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbdfc53-0624-421c-82e0-6b083865f6fd-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.662064 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.662039    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.662249 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.662230    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.716100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.716063    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.716280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.716138    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsh7g\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-kube-api-access-jsh7g\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:44.851285 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:44.851197    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:45.293407 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:45.293377    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"]
Nov 02 17:40:45.295516 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:40:45.295492    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbdfc53_0624_421c_82e0_6b083865f6fd.slice/crio-6e791d626bba140c0f51d0eb429e70a7f584b6c1101987c889f978936cfe67d2 WatchSource:0}: Error finding container 6e791d626bba140c0f51d0eb429e70a7f584b6c1101987c889f978936cfe67d2: Status 404 returned error can't find the container with id 6e791d626bba140c0f51d0eb429e70a7f584b6c1101987c889f978936cfe67d2
Nov 02 17:40:45.297806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:45.297775    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:45.297915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:45.297856    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:45.297915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:45.297904    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:46.087220 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.087179    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w" podUID="3bf50acf-3602-4f70-bdd3-2d476313c6af" containerName="payload-processing" containerID="cri-o://5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477" gracePeriod=25
Nov 02 17:40:46.153969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.153929    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" event={"ID":"2bbdfc53-0624-421c-82e0-6b083865f6fd","Type":"ContainerStarted","Data":"975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb"}
Nov 02 17:40:46.154160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.153987    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" event={"ID":"2bbdfc53-0624-421c-82e0-6b083865f6fd","Type":"ContainerStarted","Data":"6e791d626bba140c0f51d0eb429e70a7f584b6c1101987c889f978936cfe67d2"}
Nov 02 17:40:46.371139 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.371068    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podStartSLOduration=2.371047275 podStartE2EDuration="2.371047275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:46.318891469 +0000 UTC m=+1076.575922088" watchObservedRunningTime="2026-11-02 17:40:46.371047275 +0000 UTC m=+1076.628077896"
Nov 02 17:40:46.392012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.391990    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:46.480987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.480957    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-kube-api-access-nddc2\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-kube-api-access-nddc2\") pod \"3bf50acf-3602-4f70-bdd3-2d476313c6af\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") "
Nov 02 17:40:46.481179 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.481140    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-provider-credentials\") pod \"3bf50acf-3602-4f70-bdd3-2d476313c6af\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") "
Nov 02 17:40:46.481240 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.481199    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-plugins-config-volume\") pod \"3bf50acf-3602-4f70-bdd3-2d476313c6af\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") "
Nov 02 17:40:46.481283 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.481239    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-routing-overlay-volume\") pod \"3bf50acf-3602-4f70-bdd3-2d476313c6af\" (UID: \"3bf50acf-3602-4f70-bdd3-2d476313c6af\") "
Nov 02 17:40:46.481663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.481636    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-routing-overlay-volume" pod "3bf50acf-3602-4f70-bdd3-2d476313c6af" (UID: "3bf50acf-3602-4f70-bdd3-2d476313c6af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:40:46.481663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.481647    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-plugins-config-volume" pod "3bf50acf-3602-4f70-bdd3-2d476313c6af" (UID: "3bf50acf-3602-4f70-bdd3-2d476313c6af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:40:46.483549 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.483527    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-provider-credentials" pod "3bf50acf-3602-4f70-bdd3-2d476313c6af" (UID: "3bf50acf-3602-4f70-bdd3-2d476313c6af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:40:46.483680 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.483643    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-kube-api-access-nddc2" pod "3bf50acf-3602-4f70-bdd3-2d476313c6af" (UID: "3bf50acf-3602-4f70-bdd3-2d476313c6af"). InnerVolumeSpecName "kube-api-access-nddc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:40:46.582674 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.582634    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddc2\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-kube-api-access-nddc2\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:40:46.582674 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.582664    4956 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3bf50acf-3602-4f70-bdd3-2d476313c6af-provider-credentials\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:40:46.582674 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.582674    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:40:46.582904 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.582684    4956 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3bf50acf-3602-4f70-bdd3-2d476313c6af-routing-overlay-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:40:46.852024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.851981    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:46.853908 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.853885    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:46.854037 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:46.853930    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:40:47.161772 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.161684    4956 generic.go:356] "Generic (PLEG): container finished" podID="3bf50acf-3602-4f70-bdd3-2d476313c6af" containerID="5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477" exitCode=0
Nov 02 17:40:47.161772 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.161728    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"
Nov 02 17:40:47.162288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.161764    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w" event={"ID":"3bf50acf-3602-4f70-bdd3-2d476313c6af","Type":"ContainerDied","Data":"5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477"}
Nov 02 17:40:47.162288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.161819    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w" event={"ID":"3bf50acf-3602-4f70-bdd3-2d476313c6af","Type":"ContainerDied","Data":"0377b2998129561675e41589f82733dfc162adcf42d90b86397723569a604051"}
Nov 02 17:40:47.162288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.161846    4956 scope.go:122] "RemoveContainer" containerID="5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477"
Nov 02 17:40:47.172017 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.171996    4956 scope.go:122] "RemoveContainer" containerID="5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477"
Nov 02 17:40:47.172315 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:40:47.172297    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477\": container with ID starting with 5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477 not found: ID does not exist" containerID="5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477"
Nov 02 17:40:47.172397 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.172322    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477"} err="failed to get container status \"5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477\": rpc error: code = NotFound desc = could not find container \"5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477\": container with ID starting with 5149cd404f0c065d66476b92d7b30a3132700417ee6f8aee18b7d939e1aba477 not found: ID does not exist"
Nov 02 17:40:47.323872 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.323827    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"]
Nov 02 17:40:47.350206 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.348996    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-6qz4w"]
Nov 02 17:40:47.851579 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.851543    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:47.851807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.851601    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:40:47.865491 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:47.865462    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf50acf-3602-4f70-bdd3-2d476313c6af" path="/var/lib/kubelet/pods/3bf50acf-3602-4f70-bdd3-2d476313c6af/volumes"
Nov 02 17:40:48.851624 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:48.851585    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:48.852162 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:48.851653    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:40:49.851881 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:49.851837    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:49.852322 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:49.851909    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:40:50.851966 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:50.851929    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:50.852546 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:50.851992    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:40:51.851715 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:51.851673    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:51.851886 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:51.851732    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:40:52.857695 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:52.857653    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:52.858571 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:52.858544    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:52.859250 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:40:52.859226    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:40:59.056146 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:40:59.056077    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:41:29.053429 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:41:29.053389    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:41:29.961929 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:29.961844    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8cpmp-lz7d5"]
Nov 02 17:41:29.962463 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:29.962444    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3bf50acf-3602-4f70-bdd3-2d476313c6af" containerName="payload-processing"
Nov 02 17:41:29.962569 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:29.962465    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf50acf-3602-4f70-bdd3-2d476313c6af" containerName="payload-processing"
Nov 02 17:41:29.962658 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:29.962644    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf50acf-3602-4f70-bdd3-2d476313c6af" containerName="payload-processing"
Nov 02 17:41:29.966176 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:29.966156    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.020978 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.020943    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8cpmp-lz7d5"]
Nov 02 17:41:30.143675 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.143636    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-service-ca-bundle\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.144068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.143750    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqft\" (UniqueName: \"kubernetes.io/projected/8b5f1b59-868e-4445-89ba-916316d32c0c-kube-api-access-qqqft\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.144068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.143779    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-trusted-ca-bundle\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.144068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.143951    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b5f1b59-868e-4445-89ba-916316d32c0c-archives-path\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.244986 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.244896    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqft\" (UniqueName: \"kubernetes.io/projected/8b5f1b59-868e-4445-89ba-916316d32c0c-kube-api-access-qqqft\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.244986 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.244934    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-trusted-ca-bundle\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.245206 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.245056    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b5f1b59-868e-4445-89ba-916316d32c0c-archives-path\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.245206 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.245103    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-service-ca-bundle\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.245534 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.245515    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b5f1b59-868e-4445-89ba-916316d32c0c-archives-path\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.245894 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.245871    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-service-ca-bundle\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.245925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.245909    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-trusted-ca-bundle\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.302109 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.302076    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqft\" (UniqueName: \"kubernetes.io/projected/8b5f1b59-868e-4445-89ba-916316d32c0c-kube-api-access-qqqft\") pod \"periodic-gathering-8cpmp-lz7d5\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") " pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.578101 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.578014    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:41:30.952463 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:30.952430    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8cpmp-lz7d5"]
Nov 02 17:41:30.954190 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:41:30.954161    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5f1b59_868e_4445_89ba_916316d32c0c.slice/crio-16c5ca90b7bf06f869c992788fe43584c1cdc77349f441bd1d97cfb970855be3 WatchSource:0}: Error finding container 16c5ca90b7bf06f869c992788fe43584c1cdc77349f441bd1d97cfb970855be3: Status 404 returned error can't find the container with id 16c5ca90b7bf06f869c992788fe43584c1cdc77349f441bd1d97cfb970855be3
Nov 02 17:41:31.478912 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:31.478875    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5" event={"ID":"8b5f1b59-868e-4445-89ba-916316d32c0c","Type":"ContainerStarted","Data":"6831ded10f960e5ffec68dc410a25adfc76d3802166b395688e384904d21b220"}
Nov 02 17:41:31.479357 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:31.478936    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5" event={"ID":"8b5f1b59-868e-4445-89ba-916316d32c0c","Type":"ContainerStarted","Data":"16c5ca90b7bf06f869c992788fe43584c1cdc77349f441bd1d97cfb970855be3"}
Nov 02 17:41:31.609788 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:31.609733    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5" podStartSLOduration=2.60971509 podStartE2EDuration="2.60971509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:31.608164429 +0000 UTC m=+1121.865195046" watchObservedRunningTime="2026-11-02 17:41:31.60971509 +0000 UTC m=+1121.866745711"
Nov 02 17:41:42.292611 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.292579    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"]
Nov 02 17:41:42.296361 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.296344    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.348095 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.348059    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\""
Nov 02 17:41:42.374735 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.374697    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.374925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.374746    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.374925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.374775    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.375020 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.374928    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3560d50a-7609-4c5b-99f8-d094f0cccc50-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.375020 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.374994    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.375105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.375021    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.375105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.375092    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.375236 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.375221    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw669\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-kube-api-access-tw669\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.375271 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.375253    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.400436 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.400409    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"]
Nov 02 17:41:42.476354 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476319    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.476521 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476371    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.476521 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476421    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3560d50a-7609-4c5b-99f8-d094f0cccc50-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.476521 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476469    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.476521 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476507    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.476706 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476639    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.476897 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476810    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw669\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-kube-api-access-tw669\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.476897 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476859    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.477149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476909    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.477149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476927    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.477149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.476957    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.477149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.477077    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.477443 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.477420    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3560d50a-7609-4c5b-99f8-d094f0cccc50-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.477505 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.477481    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.479080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.479048    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.479190 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.479171    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.533004 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.532975    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.533233 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.533211    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw669\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-kube-api-access-tw669\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:42.609654 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:42.609620    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:43.035849 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.035816    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"]
Nov 02 17:41:43.037034 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:41:43.037012    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3560d50a_7609_4c5b_99f8_d094f0cccc50.slice/crio-9c1d1dc2c1199c5968bec0ff907a88ebe42e35b5f94c23e32ed5f1e037da36e9 WatchSource:0}: Error finding container 9c1d1dc2c1199c5968bec0ff907a88ebe42e35b5f94c23e32ed5f1e037da36e9: Status 404 returned error can't find the container with id 9c1d1dc2c1199c5968bec0ff907a88ebe42e35b5f94c23e32ed5f1e037da36e9
Nov 02 17:41:43.039380 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.039347    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:43.039484 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.039422    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:43.039553 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.039511    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:43.546701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.546657    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" event={"ID":"3560d50a-7609-4c5b-99f8-d094f0cccc50","Type":"ContainerStarted","Data":"8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee"}
Nov 02 17:41:43.547218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.546730    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" event={"ID":"3560d50a-7609-4c5b-99f8-d094f0cccc50","Type":"ContainerStarted","Data":"9c1d1dc2c1199c5968bec0ff907a88ebe42e35b5f94c23e32ed5f1e037da36e9"}
Nov 02 17:41:43.610302 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.610267    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:43.687926 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:43.687875    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podStartSLOduration=1.6878598569999999 podStartE2EDuration="1.687859857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:43.684941114 +0000 UTC m=+1133.941971736" watchObservedRunningTime="2026-11-02 17:41:43.687859857 +0000 UTC m=+1133.944890481"
Nov 02 17:41:44.612003 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:44.611965    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:41:44.612410 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:44.612034    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 17:41:45.610916 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:45.610886    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:41:45.611080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:45.610944    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 17:41:46.610522 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:46.610489    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:41:46.610948 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:46.610555    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 17:41:47.610908 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:47.610871    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:41:47.611365 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:47.610934    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 17:41:48.610020 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:48.609979    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:41:48.610229 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:48.610044    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 17:41:49.609996 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:49.609958    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:41:49.610468 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:49.610028    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 17:41:50.615183 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:50.615149    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:50.615584 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:50.615435    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:50.616225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:50.616209    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:41:51.733915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.733882    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"]
Nov 02 17:41:51.737991 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.737972    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:51.840543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.840505    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"distinct-2-w4-4ad655-kserve-self-signed-certs\""
Nov 02 17:41:51.842935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.842912    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"]
Nov 02 17:41:51.904648 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.904610    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-model-cache\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:51.904835 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.904656    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-tmp-dir\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:51.904835 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.904738    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-dshm\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:51.904835 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.904816    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c796a0b-d39e-45fd-86ac-13805db52493-tls-certs\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:51.904996 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.904846    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-home\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:51.905093 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:51.905061    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdgx\" (UniqueName: \"kubernetes.io/projected/3c796a0b-d39e-45fd-86ac-13805db52493-kube-api-access-gwdgx\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006077 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.005987    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c796a0b-d39e-45fd-86ac-13805db52493-tls-certs\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006077 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006043    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-home\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006203    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdgx\" (UniqueName: \"kubernetes.io/projected/3c796a0b-d39e-45fd-86ac-13805db52493-kube-api-access-gwdgx\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006256    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-model-cache\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006294    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-tmp-dir\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006337    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-dshm\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006575 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006554    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-home\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006658 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006637    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-model-cache\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.006790 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.006767    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-tmp-dir\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.008704 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.008676    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-dshm\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.009074 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.009058    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c796a0b-d39e-45fd-86ac-13805db52493-tls-certs\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.063066 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.063033    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdgx\" (UniqueName: \"kubernetes.io/projected/3c796a0b-d39e-45fd-86ac-13805db52493-kube-api-access-gwdgx\") pod \"distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") " pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.350446 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.350363    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:52.725494 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:52.725469    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"]
Nov 02 17:41:52.728113 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:41:52.728077    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c796a0b_d39e_45fd_86ac_13805db52493.slice/crio-6f652284f3f3804705e46ed37c452ecc53a57f844d542106071ebe4d2b1b123e WatchSource:0}: Error finding container 6f652284f3f3804705e46ed37c452ecc53a57f844d542106071ebe4d2b1b123e: Status 404 returned error can't find the container with id 6f652284f3f3804705e46ed37c452ecc53a57f844d542106071ebe4d2b1b123e
Nov 02 17:41:53.607701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:53.607662    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" event={"ID":"3c796a0b-d39e-45fd-86ac-13805db52493","Type":"ContainerStarted","Data":"da218d1952897fb3097644030ae910c86fcdf614a4f5cdbe88b50593049cb7b4"}
Nov 02 17:41:53.608142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:53.607712    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" event={"ID":"3c796a0b-d39e-45fd-86ac-13805db52493","Type":"ContainerStarted","Data":"6f652284f3f3804705e46ed37c452ecc53a57f844d542106071ebe4d2b1b123e"}
Nov 02 17:41:53.722514 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:53.722459    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" podStartSLOduration=2.722442386 podStartE2EDuration="2.722442386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:53.72032928 +0000 UTC m=+1143.977359901" watchObservedRunningTime="2026-11-02 17:41:53.722442386 +0000 UTC m=+1143.979473006"
Nov 02 17:41:54.351290 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:41:54.351243    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:41:59.056876 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:41:59.056827    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:42:02.350508 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:02.350477    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:42:02.365725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:02.365695    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:42:02.693234 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:02.693148    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:42:29.053045 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:42:29.053010    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:42:30.355586 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.355556    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"]
Nov 02 17:42:30.360181 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.360155    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.456461 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.456425    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"]
Nov 02 17:42:30.513485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513453    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513497    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513529    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8qj\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-kube-api-access-sh8qj\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513608    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513676 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513658    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513721    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513757    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513785    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.513976 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.513926    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615386 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615289    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615407    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615449    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615471    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8qj\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-kube-api-access-sh8qj\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615496    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615527    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615558    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615592    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615636    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615703    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615763    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.615879 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615817    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.616108 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.615892    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.616290 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.616272    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.618070 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.618044    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.618284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.618257    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.672023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.671993    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.672278 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.672255    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8qj\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-kube-api-access-sh8qj\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:30.674933 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:30.674910    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:31.104762 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:31.104728    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"]
Nov 02 17:42:31.105974 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:42:31.105950    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491d7fbc_9e54_48c8_b5aa_17bb0a948eed.slice/crio-284fa93a452aa393d6f69df1c479914a3b1f9eb1c4ab17701863f8b54a43a2f7 WatchSource:0}: Error finding container 284fa93a452aa393d6f69df1c479914a3b1f9eb1c4ab17701863f8b54a43a2f7: Status 404 returned error can't find the container with id 284fa93a452aa393d6f69df1c479914a3b1f9eb1c4ab17701863f8b54a43a2f7
Nov 02 17:42:31.108320 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:31.108289    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:31.108395 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:31.108352    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:31.108395 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:31.108381    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:31.864971 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:31.864935    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" event={"ID":"491d7fbc-9e54-48c8-b5aa-17bb0a948eed","Type":"ContainerStarted","Data":"a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981"}
Nov 02 17:42:31.865374 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:31.864988    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" event={"ID":"491d7fbc-9e54-48c8-b5aa-17bb0a948eed","Type":"ContainerStarted","Data":"284fa93a452aa393d6f69df1c479914a3b1f9eb1c4ab17701863f8b54a43a2f7"}
Nov 02 17:42:32.097227 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:32.097167    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podStartSLOduration=2.097152346 podStartE2EDuration="2.097152346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:32.047494405 +0000 UTC m=+1182.304525030" watchObservedRunningTime="2026-11-02 17:42:32.097152346 +0000 UTC m=+1182.354182975"
Nov 02 17:42:32.675539 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:32.675493    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:32.677385 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:32.677354    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:42:32.677531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:32.677413    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 17:42:33.676282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:33.676240    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:42:33.676687 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:33.676303    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 17:42:34.675533 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:34.675496    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:42:34.675759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:34.675572    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 17:42:35.676297 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:35.676254    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:42:35.676826 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:35.676343    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 17:42:36.676361 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:36.676326    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:42:36.676776 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:36.676387    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 17:42:37.675984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:37.675935    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:42:37.676211 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:37.676008    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 17:42:38.679974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:38.679946    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:38.680468 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:38.680363    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:38.681000 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:38.680983    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:42:49.991757 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:49.991728    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:42:49.992234 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:42:49.992187    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:42:59.053997 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:42:59.053960    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:43:08.083024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:08.082988    4956 generic.go:356] "Generic (PLEG): container finished" podID="8b5f1b59-868e-4445-89ba-916316d32c0c" containerID="6831ded10f960e5ffec68dc410a25adfc76d3802166b395688e384904d21b220" exitCode=0
Nov 02 17:43:08.083669 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:08.083061    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5" event={"ID":"8b5f1b59-868e-4445-89ba-916316d32c0c","Type":"ContainerDied","Data":"6831ded10f960e5ffec68dc410a25adfc76d3802166b395688e384904d21b220"}
Nov 02 17:43:09.239836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.239808    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:43:09.417424 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.417267    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-service-ca-bundle\") pod \"8b5f1b59-868e-4445-89ba-916316d32c0c\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") "
Nov 02 17:43:09.417646 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.417470    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-trusted-ca-bundle\") pod \"8b5f1b59-868e-4445-89ba-916316d32c0c\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") "
Nov 02 17:43:09.417646 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.417559    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5f1b59-868e-4445-89ba-916316d32c0c-kube-api-access-qqqft\" (UniqueName: \"kubernetes.io/projected/8b5f1b59-868e-4445-89ba-916316d32c0c-kube-api-access-qqqft\") pod \"8b5f1b59-868e-4445-89ba-916316d32c0c\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") "
Nov 02 17:43:09.417646 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.417584    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b5f1b59-868e-4445-89ba-916316d32c0c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b5f1b59-868e-4445-89ba-916316d32c0c-archives-path\") pod \"8b5f1b59-868e-4445-89ba-916316d32c0c\" (UID: \"8b5f1b59-868e-4445-89ba-916316d32c0c\") "
Nov 02 17:43:09.417877 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.417788    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-service-ca-bundle" pod "8b5f1b59-868e-4445-89ba-916316d32c0c" (UID: "8b5f1b59-868e-4445-89ba-916316d32c0c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:09.418004 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.417980    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5f1b59-868e-4445-89ba-916316d32c0c-archives-path" pod "8b5f1b59-868e-4445-89ba-916316d32c0c" (UID: "8b5f1b59-868e-4445-89ba-916316d32c0c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:43:09.418053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.418007    4956 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-service-ca-bundle\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:43:09.418053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.417984    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-trusted-ca-bundle" pod "8b5f1b59-868e-4445-89ba-916316d32c0c" (UID: "8b5f1b59-868e-4445-89ba-916316d32c0c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:09.419983 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.419964    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5f1b59-868e-4445-89ba-916316d32c0c-kube-api-access-qqqft" pod "8b5f1b59-868e-4445-89ba-916316d32c0c" (UID: "8b5f1b59-868e-4445-89ba-916316d32c0c"). InnerVolumeSpecName "kube-api-access-qqqft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:09.518866 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.518829    4956 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b5f1b59-868e-4445-89ba-916316d32c0c-trusted-ca-bundle\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:43:09.518866 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.518861    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqft\" (UniqueName: \"kubernetes.io/projected/8b5f1b59-868e-4445-89ba-916316d32c0c-kube-api-access-qqqft\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:43:09.518866 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:09.518872    4956 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b5f1b59-868e-4445-89ba-916316d32c0c-archives-path\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:43:10.095633 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:10.095600    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5"
Nov 02 17:43:10.095817 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:10.095638    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8cpmp-lz7d5" event={"ID":"8b5f1b59-868e-4445-89ba-916316d32c0c","Type":"ContainerDied","Data":"16c5ca90b7bf06f869c992788fe43584c1cdc77349f441bd1d97cfb970855be3"}
Nov 02 17:43:10.095817 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:10.095695    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c5ca90b7bf06f869c992788fe43584c1cdc77349f441bd1d97cfb970855be3"
Nov 02 17:43:28.109183 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.109136    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"]
Nov 02 17:43:28.109885 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.109870    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8b5f1b59-868e-4445-89ba-916316d32c0c" containerName="insights-gathering"
Nov 02 17:43:28.109930 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.109887    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5f1b59-868e-4445-89ba-916316d32c0c" containerName="insights-gathering"
Nov 02 17:43:28.110083 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.110073    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5f1b59-868e-4445-89ba-916316d32c0c" containerName="insights-gathering"
Nov 02 17:43:28.115226 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.115203    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153294 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153248    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153312    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153471 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153437    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cade3928-f203-4e74-8998-d541afacb9a5-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153564 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153486    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7db6x\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-kube-api-access-7db6x\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153608 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153579    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cade3928-f203-4e74-8998-d541afacb9a5-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153649 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153610    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153704 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153653    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153755 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153707    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.153806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.153783    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.171800 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.171773    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"]
Nov 02 17:43:28.255107 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255070    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255107 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255110    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255182    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255214    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255280    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cade3928-f203-4e74-8998-d541afacb9a5-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255306    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db6x\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-kube-api-access-7db6x\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255364    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cade3928-f203-4e74-8998-d541afacb9a5-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255408 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255385    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255437    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.255730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.255507    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.256370 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.256104    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.256370 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.256184    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.256370 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.256277    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.256596 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.256484    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cade3928-f203-4e74-8998-d541afacb9a5-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.258430 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.258406    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.258666 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.258644    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cade3928-f203-4e74-8998-d541afacb9a5-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.311903 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.311863    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.312064 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.311929    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7db6x\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-kube-api-access-7db6x\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.429492 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.429391    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:28.858921 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.858896    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"]
Nov 02 17:43:28.859991 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:43:28.859968    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcade3928_f203_4e74_8998_d541afacb9a5.slice/crio-5b681ed9b0b2d735bff546b3788f58240662f9f0081e43f1fd06cbda8f9ef42a WatchSource:0}: Error finding container 5b681ed9b0b2d735bff546b3788f58240662f9f0081e43f1fd06cbda8f9ef42a: Status 404 returned error can't find the container with id 5b681ed9b0b2d735bff546b3788f58240662f9f0081e43f1fd06cbda8f9ef42a
Nov 02 17:43:28.862218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.862185    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:28.862314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.862250    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:28.862314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:28.862292    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:29.059827 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:43:29.059789    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:43:29.236326 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:29.236283    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" event={"ID":"cade3928-f203-4e74-8998-d541afacb9a5","Type":"ContainerStarted","Data":"b1fa5362a20ea725bfa8f1ffa2cca41316204b5e916f06785b1211f11ff89a7e"}
Nov 02 17:43:29.236801 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:29.236357    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" event={"ID":"cade3928-f203-4e74-8998-d541afacb9a5","Type":"ContainerStarted","Data":"5b681ed9b0b2d735bff546b3788f58240662f9f0081e43f1fd06cbda8f9ef42a"}
Nov 02 17:43:29.408001 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:29.407933    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podStartSLOduration=1.40791563 podStartE2EDuration="1.40791563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:29.355578645 +0000 UTC m=+1239.612609267" watchObservedRunningTime="2026-11-02 17:43:29.40791563 +0000 UTC m=+1239.664946252"
Nov 02 17:43:29.432153 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:29.430483    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:29.440165 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:29.438314    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:29.440165 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:29.438412    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 17:43:30.450168 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.447521    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:30.450168 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.447580    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 17:43:30.809807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.809775    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"]
Nov 02 17:43:30.813701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.813679    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888268    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888505 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888391    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888505 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888490    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888602 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888520    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhl5k\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-kube-api-access-bhl5k\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888655    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888755 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888705    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888754    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888794    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.888903 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.888871    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.917164 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.917079    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"]
Nov 02 17:43:30.990231 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990191    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990274    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990300    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhl5k\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-kube-api-access-bhl5k\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990347    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990386    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990442 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990443    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990729 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990502    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990729 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990579    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990729 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990677    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.990938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.990890    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.991051 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.991022    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.991218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.991109    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.991372 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.991351    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.991449 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.991432    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.992923 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.992899    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:30.993312 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:30.993295    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:31.053154 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.053095    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:31.053567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.053542    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhl5k\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-kube-api-access-bhl5k\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:31.153573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.153498    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:31.450586 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.450553    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:31.451010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.450622    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 17:43:31.581252 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.581226    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"]
Nov 02 17:43:31.582467 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:43:31.582442    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b939f8_20cf_4b13_8d25_b0ad0ba8c9f5.slice/crio-718ef2920b015569ae92c46e18c622ec6e61fa5f65a144bd43ebcad94fa93ce1 WatchSource:0}: Error finding container 718ef2920b015569ae92c46e18c622ec6e61fa5f65a144bd43ebcad94fa93ce1: Status 404 returned error can't find the container with id 718ef2920b015569ae92c46e18c622ec6e61fa5f65a144bd43ebcad94fa93ce1
Nov 02 17:43:31.584931 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.584901    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:31.585009 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.584961    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:31.585009 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:31.584988    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:32.275303 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:32.275262    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" event={"ID":"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5","Type":"ContainerStarted","Data":"5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e"}
Nov 02 17:43:32.275474 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:32.275363    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" event={"ID":"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5","Type":"ContainerStarted","Data":"718ef2920b015569ae92c46e18c622ec6e61fa5f65a144bd43ebcad94fa93ce1"}
Nov 02 17:43:32.448148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:32.447264    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:32.448148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:32.447341    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 17:43:32.476293 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:32.474432    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podStartSLOduration=2.4744115239999998 podStartE2EDuration="2.474411524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.472615027 +0000 UTC m=+1242.729645679" watchObservedRunningTime="2026-11-02 17:43:32.474411524 +0000 UTC m=+1242.731442145"
Nov 02 17:43:33.160147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:33.156289    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:33.160147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:33.158225    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:33.160147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:33.158301    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:33.477302 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:33.465258    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:33.477302 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:33.465326    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 17:43:34.156148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:34.155570    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:34.156148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:34.155634    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:34.434740 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:34.434644    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:34.435047 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:34.435002    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 17:43:35.154541 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:35.154505    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:35.155045 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:35.154580    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:35.438202 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:35.435544    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:35.438202 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:35.435610    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 17:43:36.156519 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.156481    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:36.156938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.156543    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:36.468427 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.468388    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:36.468765 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.468739    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:36.470642 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.470619    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:43:36.858763 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.858677    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"]
Nov 02 17:43:36.862784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.862760    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:36.902560 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:36.902525    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"]
Nov 02 17:43:37.007378 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.007343    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.007598 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.007448    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.007598 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.007502    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.007598 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.007566    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.007777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.007760    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438add23-9e68-4bf7-a253-84ae0bcece58-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.007837 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.007817    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccz6\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-kube-api-access-wccz6\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.008052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.008024    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438add23-9e68-4bf7-a253-84ae0bcece58-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.008167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.008066    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.008262 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.008244    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109565 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109473    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109565 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109517    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109578    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438add23-9e68-4bf7-a253-84ae0bcece58-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109605    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccz6\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-kube-api-access-wccz6\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109680    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438add23-9e68-4bf7-a253-84ae0bcece58-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109707    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109802    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109855    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109900    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109915    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.109992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.109945    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.110297 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.110253    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.110374 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.110355    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.110520 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.110498    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438add23-9e68-4bf7-a253-84ae0bcece58-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.112193 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.112174    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.112291 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.112247    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438add23-9e68-4bf7-a253-84ae0bcece58-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.154787 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.154741    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:37.155003 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.154799    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:37.167058 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.167032    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.167474 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.167247    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccz6\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-kube-api-access-wccz6\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.179717 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.179691    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:37.605419 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.605387    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"]
Nov 02 17:43:37.610174 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.610154    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.681230 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.681198    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"]
Nov 02 17:43:37.682968 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:43:37.682940    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438add23_9e68_4bf7_a253_84ae0bcece58.slice/crio-8f998fe6c520a0028b6b70268f301c67baa337ae0b85b670022b33d56ea03496 WatchSource:0}: Error finding container 8f998fe6c520a0028b6b70268f301c67baa337ae0b85b670022b33d56ea03496: Status 404 returned error can't find the container with id 8f998fe6c520a0028b6b70268f301c67baa337ae0b85b670022b33d56ea03496
Nov 02 17:43:37.686369 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.686327    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:37.686469 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.686417    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:37.686507 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.686464    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:37.716996 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.716961    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717024    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717195    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrlwr\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-kube-api-access-vrlwr\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717236    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717319    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717357    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717500    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717549    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1abc52b-f034-464d-87d5-ceb44b55255d-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.717603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.717597    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.720352 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.720201    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"]
Nov 02 17:43:37.755871 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.755838    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6e3175-openshift-default-dockercfg-n7khr\""
Nov 02 17:43:37.824823 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.824790    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.824823 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.824834    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1abc52b-f034-464d-87d5-ceb44b55255d-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825055 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825018    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825240 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825218    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825474 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825448    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825595 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825512    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlwr\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-kube-api-access-vrlwr\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825595 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825516    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1abc52b-f034-464d-87d5-ceb44b55255d-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825595 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825545    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825773 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825755    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825830 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825804    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.825882 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.825831    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.826247 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.826214    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.826462 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.826441    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.826527 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.826495    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.828355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.828331    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.828791 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.828767    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.882732 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.882657    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlwr\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-kube-api-access-vrlwr\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.883190 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.883163    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:37.924087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:37.924050    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:38.154531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.154431    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:38.154531 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.154500    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:38.318336 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.318302    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" event={"ID":"438add23-9e68-4bf7-a253-84ae0bcece58","Type":"ContainerStarted","Data":"b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac"}
Nov 02 17:43:38.318858 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.318361    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" event={"ID":"438add23-9e68-4bf7-a253-84ae0bcece58","Type":"ContainerStarted","Data":"8f998fe6c520a0028b6b70268f301c67baa337ae0b85b670022b33d56ea03496"}
Nov 02 17:43:38.360432 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:43:38.360376    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1abc52b_f034_464d_87d5_ceb44b55255d.slice/crio-add5a00d255e9debdc249c17aaad38ec06cdf181d3b0b75f5ea707090ce8f6d8 WatchSource:0}: Error finding container add5a00d255e9debdc249c17aaad38ec06cdf181d3b0b75f5ea707090ce8f6d8: Status 404 returned error can't find the container with id add5a00d255e9debdc249c17aaad38ec06cdf181d3b0b75f5ea707090ce8f6d8
Nov 02 17:43:38.361318 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.361204    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"]
Nov 02 17:43:38.363377 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.363337    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:38.363488 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.363424    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:38.363488 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.363468    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:38.437576 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:38.437341    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podStartSLOduration=2.43732794 podStartE2EDuration="2.43732794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:38.435621329 +0000 UTC m=+1248.692651949" watchObservedRunningTime="2026-11-02 17:43:38.43732794 +0000 UTC m=+1248.694358559"
Nov 02 17:43:39.182026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.181994    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:39.182443 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.182398    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:39.182615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.182588    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:39.291224 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.288650    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:39.291224 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.290269    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:39.297454 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.297382    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 17:43:39.325328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.325291    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" event={"ID":"e1abc52b-f034-464d-87d5-ceb44b55255d","Type":"ContainerStarted","Data":"87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0"}
Nov 02 17:43:39.325809 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.325351    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" event={"ID":"e1abc52b-f034-464d-87d5-ceb44b55255d","Type":"ContainerStarted","Data":"add5a00d255e9debdc249c17aaad38ec06cdf181d3b0b75f5ea707090ce8f6d8"}
Nov 02 17:43:39.924417 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.924383    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:39.976192 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.975292    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:39.976192 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:39.975350    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:40.181641 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:40.181509    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:40.181641 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:40.181587    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:40.947144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:40.939246    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:40.947144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:40.939332    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:41.180652 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:41.180619    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:41.180823 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:41.180674    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:41.924479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:41.924444    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:41.924650 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:41.924501    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:42.181197 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:42.180656    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:42.181197 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:42.180713    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:42.925113 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:42.925078    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:42.925351 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:42.925163    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:43.181250 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:43.180561    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:43.181250 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:43.180630    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:43.932149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:43.924523    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:43.932149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:43.924585    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:44.224202 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.207766    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:44.224202 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.207825    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:44.687207 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.685216    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:44.732146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.731997    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:44.751149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.741599    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:44.927144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.923386    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:44.927144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.924790    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:44.927144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.924836    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:45.006147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:44.999257    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.182149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.181590    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:45.182149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.181671    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:45.265144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.265026    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.416147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.407228    4956 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.416147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.410314    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct" podUID="5e711d1c-6058-4345-9189-de8749615f2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.472176 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.463249    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.826149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.721006    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.826149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.721786    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:43:45.826149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.732263    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.826149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.772266    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:43:45.826149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.814806    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:43:45.826149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.815231    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:43:45.890148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.876175    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podStartSLOduration=8.876154833 podStartE2EDuration="8.876154833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:39.773538458 +0000 UTC m=+1250.030569081" watchObservedRunningTime="2026-11-02 17:43:45.876154833 +0000 UTC m=+1256.133185453"
Nov 02 17:43:45.955324 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.931707    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.955324 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.934747    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:45.955324 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:45.934796    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:46.045923 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.045875    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:46.140208 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.132864    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:43:46.140557 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.140529    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:43:46.187748 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.187715    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:46.188009 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.187984    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:46.219131 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.219088    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 17:43:46.232664 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.232607    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 17:43:46.283830 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.283780    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:46.284644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.284611    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:43:46.437201 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.436267    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct" podUID="5e711d1c-6058-4345-9189-de8749615f2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:46.445281 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.444958    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:43:46.466650 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.465243    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:46.473375 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.473343    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:43:46.525145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.511517    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 17:43:46.533976 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.533949    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct"
Nov 02 17:43:46.557602 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.557568    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 17:43:46.722905 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.722876    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 17:43:46.742584 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.724555    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 17:43:46.746282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.746237    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:46.770789 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.770759    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 17:43:46.926784 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.926750    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:43:46.927050 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:46.927022    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 02 17:43:47.931270 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:47.931239    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:47.931709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:47.931662    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:47.932600 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:47.932575    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:43:48.221325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:48.221286    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:48.221500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:48.221350    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:48.297092 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:48.297052    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:48.297517 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:48.297491    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:48.298525 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:43:48.298507    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:43:59.052837 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:43:59.052793    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:44:06.234144 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:06.233339    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:06.517325 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:06.517211    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:06.540231 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:06.540181    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct" podUID="5e711d1c-6058-4345-9189-de8749615f2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 02 17:44:06.585146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:06.578602    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:06.727269 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:06.727213    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:06.727921 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:06.727889    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:06.775369 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:06.775257    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:11.577771 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:11.577738    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"]
Nov 02 17:44:11.578268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:11.577937    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy" containerID="cri-o://50e3dde6544f39e217acaf6b8237fddb8f74b0ff59eb29443d1986b1a32c6986" gracePeriod=30
Nov 02 17:44:13.476063 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.476026    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"]
Nov 02 17:44:13.481193 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.481177    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585093    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585228    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585259    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585440 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585423    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585493 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585466    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7cq\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-kube-api-access-5w7cq\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585542 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585492    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585655 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585629    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844b2aa-bc8f-4996-af46-41669acac641-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585703    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.585850 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.585833    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844b2aa-bc8f-4996-af46-41669acac641-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.676105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.676069    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"]
Nov 02 17:44:13.687588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687557    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844b2aa-bc8f-4996-af46-41669acac641-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687760 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687618    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687760 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687658    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687760 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687672    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687767    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687794    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7cq\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-kube-api-access-5w7cq\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687818    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687857    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844b2aa-bc8f-4996-af46-41669acac641-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.687936 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.687912    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.688445 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.688414    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844b2aa-bc8f-4996-af46-41669acac641-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.688653 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.688623    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.688766 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.688731    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.689033 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.689009    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.689308 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.689282    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.691204 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.691181    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.691301 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.691199    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844b2aa-bc8f-4996-af46-41669acac641-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.754101 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.754029    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.754837 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.754814    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7cq\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-kube-api-access-5w7cq\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:13.795047 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:13.795021    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:14.243482 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:44:14.243450    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc844b2aa_bc8f_4996_af46_41669acac641.slice/crio-7399f75897c781a277911042e25f55bf2f22d8c0fe709e376c88f36d4abaee77 WatchSource:0}: Error finding container 7399f75897c781a277911042e25f55bf2f22d8c0fe709e376c88f36d4abaee77: Status 404 returned error can't find the container with id 7399f75897c781a277911042e25f55bf2f22d8c0fe709e376c88f36d4abaee77
Nov 02 17:44:14.243699 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:14.243679    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"]
Nov 02 17:44:14.246036 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:14.246006    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:14.246148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:14.246064    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:14.246148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:14.246093    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:14.860181 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:14.860140    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" event={"ID":"c844b2aa-bc8f-4996-af46-41669acac641","Type":"ContainerStarted","Data":"b88735322686996e0f53179b10a8dcd1d90d2043337590b13ffcb7a1709df640"}
Nov 02 17:44:14.860601 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:14.860206    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" event={"ID":"c844b2aa-bc8f-4996-af46-41669acac641","Type":"ContainerStarted","Data":"7399f75897c781a277911042e25f55bf2f22d8c0fe709e376c88f36d4abaee77"}
Nov 02 17:44:15.047708 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:15.047644    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podStartSLOduration=2.047619783 podStartE2EDuration="2.047619783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:15.024863799 +0000 UTC m=+1285.281894444" watchObservedRunningTime="2026-11-02 17:44:15.047619783 +0000 UTC m=+1285.304650403"
Nov 02 17:44:15.795584 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:15.795550    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:15.797439 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:15.797417    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 17:44:15.797504 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:15.797468    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 17:44:16.795794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:16.795760    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 17:44:16.796257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:16.795826    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 17:44:16.875620 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:16.875589    4956 generic.go:356] "Generic (PLEG): container finished" podID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerID="50e3dde6544f39e217acaf6b8237fddb8f74b0ff59eb29443d1986b1a32c6986" exitCode=0
Nov 02 17:44:16.875790 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:16.875657    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" event={"ID":"5c6f0b20-ec54-46da-94b3-99ea645767da","Type":"ContainerDied","Data":"50e3dde6544f39e217acaf6b8237fddb8f74b0ff59eb29443d1986b1a32c6986"}
Nov 02 17:44:16.901730 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:16.901706    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.063610    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-envoy\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.063690    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-data\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.063732    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-socket\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.063760    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-token\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.063828    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-certs\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.063950    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6f0b20-ec54-46da-94b3-99ea645767da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f0b20-ec54-46da-94b3-99ea645767da-istiod-ca-cert\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.064021    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-podinfo\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.064145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.064077    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-kube-api-access-wwfxz\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-kube-api-access-wwfxz\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.066034 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.064819    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-credential-socket\") pod \"5c6f0b20-ec54-46da-94b3-99ea645767da\" (UID: \"5c6f0b20-ec54-46da-94b3-99ea645767da\") "
Nov 02 17:44:17.066034 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.065654    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-credential-socket" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:17.066034 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.065868    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-certs" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:17.066290 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.066212    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-data" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:17.066832 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.066544    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6f0b20-ec54-46da-94b3-99ea645767da-istiod-ca-cert" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:17.067965 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.067932    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-socket" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:17.070016 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.069938    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-podinfo" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:44:17.071884 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.071853    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-token" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:17.072314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.072285    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-kube-api-access-wwfxz" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "kube-api-access-wwfxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:17.072488 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.072467    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-envoy" pod "5c6f0b20-ec54-46da-94b3-99ea645767da" (UID: "5c6f0b20-ec54-46da-94b3-99ea645767da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166562    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfxz\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-kube-api-access-wwfxz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166601    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166617    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166630    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166639    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166650    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166659    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f0b20-ec54-46da-94b3-99ea645767da-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.166665 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166668    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f0b20-ec54-46da-94b3-99ea645767da-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.167205 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.166676    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f0b20-ec54-46da-94b3-99ea645767da-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.300620 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.300588    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"]
Nov 02 17:44:17.301147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.301134    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy"
Nov 02 17:44:17.301206 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.301148    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy"
Nov 02 17:44:17.301282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.301273    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" containerName="istio-proxy"
Nov 02 17:44:17.304815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.304798    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:17.361162 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.361079    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8cbedb9f\""
Nov 02 17:44:17.369638 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.361750    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8cbedb9f-dockercfg-7cv2l\""
Nov 02 17:44:17.369638 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.365788    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"]
Nov 02 17:44:17.458224 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.458191    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"]
Nov 02 17:44:17.462560 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.462536    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.470635 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.470610    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhnb\" (UniqueName: \"kubernetes.io/projected/81858b78-1132-4786-a3ed-f93fd27130be-kube-api-access-wqhnb\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:17.470896 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.470866    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81858b78-1132-4786-a3ed-f93fd27130be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:17.513887 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.508175    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"]
Nov 02 17:44:17.575943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.574416    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhmf\" (UniqueName: \"kubernetes.io/projected/5344119c-53fc-4e51-8033-d3a7d4919c22-kube-api-access-ghhmf\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.575943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.574561    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5344119c-53fc-4e51-8033-d3a7d4919c22-plugins-config-volume\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.575943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.574634    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhnb\" (UniqueName: \"kubernetes.io/projected/81858b78-1132-4786-a3ed-f93fd27130be-kube-api-access-wqhnb\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:17.575943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.574671    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81858b78-1132-4786-a3ed-f93fd27130be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:17.575943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.575752    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81858b78-1132-4786-a3ed-f93fd27130be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:17.652940 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.652903    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhnb\" (UniqueName: \"kubernetes.io/projected/81858b78-1132-4786-a3ed-f93fd27130be-kube-api-access-wqhnb\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:17.676042 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.676006    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5344119c-53fc-4e51-8033-d3a7d4919c22-plugins-config-volume\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.676229 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.676211    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhmf\" (UniqueName: \"kubernetes.io/projected/5344119c-53fc-4e51-8033-d3a7d4919c22-kube-api-access-ghhmf\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.676872 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.676830    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5344119c-53fc-4e51-8033-d3a7d4919c22-plugins-config-volume\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.759594 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.759512    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhmf\" (UniqueName: \"kubernetes.io/projected/5344119c-53fc-4e51-8033-d3a7d4919c22-kube-api-access-ghhmf\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.783783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.783752    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:17.795420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.795393    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 17:44:17.795524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.795448    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 17:44:17.884756 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.884691    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"
Nov 02 17:44:17.884756 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.884709    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4" event={"ID":"5c6f0b20-ec54-46da-94b3-99ea645767da","Type":"ContainerDied","Data":"9a7981bc761c2f853c41eab90290bd39b6586f202981ab1c2def5f4df3b41850"}
Nov 02 17:44:17.885231 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.884776    4956 scope.go:122] "RemoveContainer" containerID="50e3dde6544f39e217acaf6b8237fddb8f74b0ff59eb29443d1986b1a32c6986"
Nov 02 17:44:17.922434 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:17.922396    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:18.061711 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.061676    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"]
Nov 02 17:44:18.064585 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.064535    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8pkn4"]
Nov 02 17:44:18.184570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.184541    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"]
Nov 02 17:44:18.185728 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:44:18.185704    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5344119c_53fc_4e51_8033_d3a7d4919c22.slice/crio-e9d000cfd91c767e0edde739bcd8e0dbc735964448533662d62f6fb040ea9381 WatchSource:0}: Error finding container e9d000cfd91c767e0edde739bcd8e0dbc735964448533662d62f6fb040ea9381: Status 404 returned error can't find the container with id e9d000cfd91c767e0edde739bcd8e0dbc735964448533662d62f6fb040ea9381
Nov 02 17:44:18.350208 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.348557    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"]
Nov 02 17:44:18.795779 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.795738    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 17:44:18.795959 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.795804    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 17:44:18.892500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.892467    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" event={"ID":"81858b78-1132-4786-a3ed-f93fd27130be","Type":"ContainerStarted","Data":"b9e35dc94d2bd1039bce0f22ade09a111c78d2598edd1653fef02e0d1ee6cb65"}
Nov 02 17:44:18.892949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.892527    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" event={"ID":"81858b78-1132-4786-a3ed-f93fd27130be","Type":"ContainerStarted","Data":"a11f93765c253800f46b76386f1f188760e7642dee4b916c8c32afead9848c41"}
Nov 02 17:44:18.892949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.892684    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:18.895181 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.895152    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" event={"ID":"5344119c-53fc-4e51-8033-d3a7d4919c22","Type":"ContainerStarted","Data":"76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b"}
Nov 02 17:44:18.895288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.895199    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" event={"ID":"5344119c-53fc-4e51-8033-d3a7d4919c22","Type":"ContainerStarted","Data":"e9d000cfd91c767e0edde739bcd8e0dbc735964448533662d62f6fb040ea9381"}
Nov 02 17:44:18.895288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:18.895256    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:19.012036 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:19.011982    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" podStartSLOduration=2.011969318 podStartE2EDuration="2.011969318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:19.009464135 +0000 UTC m=+1289.266494757" watchObservedRunningTime="2026-11-02 17:44:19.011969318 +0000 UTC m=+1289.268999937"
Nov 02 17:44:19.123168 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:19.123031    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" podStartSLOduration=2.123018646 podStartE2EDuration="2.123018646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:19.122479816 +0000 UTC m=+1289.379510446" watchObservedRunningTime="2026-11-02 17:44:19.123018646 +0000 UTC m=+1289.380049274"
Nov 02 17:44:19.795591 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:19.795504    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 17:44:19.795591 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:19.795555    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 17:44:19.866420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:19.866381    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6f0b20-ec54-46da-94b3-99ea645767da" path="/var/lib/kubelet/pods/5c6f0b20-ec54-46da-94b3-99ea645767da/volumes"
Nov 02 17:44:20.796292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:20.796253    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 17:44:20.796818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:20.796307    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 17:44:21.800417 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:21.800388    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:21.800807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:21.800672    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:21.801273 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:21.801257    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:44:22.995397 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:22.995366    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"]
Nov 02 17:44:23.000696 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.000675    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048305 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048271    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9dh\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-kube-api-access-8l9dh\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048400    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048485 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048472    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764791c-08d9-4913-8496-9f0f0c566e98-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048618 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048577    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048671 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048625    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048671 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048654    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048776 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048743    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048776 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048773    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.048881 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.048800    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764791c-08d9-4913-8496-9f0f0c566e98-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.084565 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.084524    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"]
Nov 02 17:44:23.149759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.149723    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764791c-08d9-4913-8496-9f0f0c566e98-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.149925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.149787    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.149925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.149834    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.149925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.149864    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.149925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.149907    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150085 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.149946    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150083    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764791c-08d9-4913-8496-9f0f0c566e98-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150240 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150217    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9dh\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-kube-api-access-8l9dh\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150373 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150344    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150373 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150366    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150407    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150482    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150862 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150829    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.150862 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.150848    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764791c-08d9-4913-8496-9f0f0c566e98-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.152416 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.152396    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.152553 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.152537    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764791c-08d9-4913-8496-9f0f0c566e98-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.206794 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.206762    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.206904 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.206845    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9dh\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-kube-api-access-8l9dh\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.314776 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.314692    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:23.747450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.747427    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"]
Nov 02 17:44:23.748680 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:44:23.748657    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode764791c_08d9_4913_8496_9f0f0c566e98.slice/crio-e6689b688a8a58d0985cb28175dabcb87651e9c7976134bee9f54d5e06a142d2 WatchSource:0}: Error finding container e6689b688a8a58d0985cb28175dabcb87651e9c7976134bee9f54d5e06a142d2: Status 404 returned error can't find the container with id e6689b688a8a58d0985cb28175dabcb87651e9c7976134bee9f54d5e06a142d2
Nov 02 17:44:23.751575 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.751537    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:23.751678 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.751620    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:23.751678 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.751663    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:23.926195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.926150    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" event={"ID":"e764791c-08d9-4913-8496-9f0f0c566e98","Type":"ContainerStarted","Data":"7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a"}
Nov 02 17:44:23.926391 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:23.926208    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" event={"ID":"e764791c-08d9-4913-8496-9f0f0c566e98","Type":"ContainerStarted","Data":"e6689b688a8a58d0985cb28175dabcb87651e9c7976134bee9f54d5e06a142d2"}
Nov 02 17:44:24.043326 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:24.043211    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podStartSLOduration=2.043190398 podStartE2EDuration="2.043190398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:24.041283785 +0000 UTC m=+1294.298314406" watchObservedRunningTime="2026-11-02 17:44:24.043190398 +0000 UTC m=+1294.300221020"
Nov 02 17:44:24.315970 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:24.315889    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:24.317515 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:24.317493    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:24.317636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:24.317537    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:25.315449 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:25.315417    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:25.315966 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:25.315470    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:26.315450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:26.315417    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:26.315681 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:26.315490    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:27.315105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:27.315071    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:27.315298 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:27.315165    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:28.315737 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:28.315706    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:28.316173 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:28.315773    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:29.075435 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:44:29.075387    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:44:29.315561 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:29.315511    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:29.315857 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:29.315829    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:29.905185 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:29.905085    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:44:29.905342 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:29.905218    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:44:30.316083 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:30.316046    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:30.316567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:30.316187    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:32.129282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:32.129253    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:32.131242 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:32.131222    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:32.134453 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:32.134432    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 17:44:33.444147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:33.443578    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:33.444147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:33.443690    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:43.682643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.682606    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"]
Nov 02 17:44:43.686710 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.686694    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.738306 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.738276    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8cbedb9f-serving-cert\""
Nov 02 17:44:43.783113 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.783086    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-metrics-certs\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.783286 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.783138    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.783328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.783276    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21c3-106e-4697-bdb2-b95a17c8a435-openshift-service-ca\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.783328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.783310    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jx8\" (UniqueName: \"kubernetes.io/projected/b25e21c3-106e-4697-bdb2-b95a17c8a435-kube-api-access-46jx8\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.790925 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.790899    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"]
Nov 02 17:44:43.884657 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.884623    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21c3-106e-4697-bdb2-b95a17c8a435-openshift-service-ca\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.884657 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.884661    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jx8\" (UniqueName: \"kubernetes.io/projected/b25e21c3-106e-4697-bdb2-b95a17c8a435-kube-api-access-46jx8\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.884859 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.884824    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-metrics-certs\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.884897 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.884855    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.884986 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:44:43.884972    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-serving-cert: secret "maas-api-e2e-mt-8cbedb9f-serving-cert" not found
Nov 02 17:44:43.885058 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:44:43.885047    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls podName:b25e21c3-106e-4697-bdb2-b95a17c8a435 nodeName:}" failed. No retries permitted until 2026-11-02 17:44:44.385028311 +0000 UTC m=+1314.642058911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls") pod "maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb" (UID: "b25e21c3-106e-4697-bdb2-b95a17c8a435") : secret "maas-api-e2e-mt-8cbedb9f-serving-cert" not found
Nov 02 17:44:43.885294 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.885272    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21c3-106e-4697-bdb2-b95a17c8a435-openshift-service-ca\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.887434 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.887412    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-metrics-certs\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:43.941948 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:43.941883    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jx8\" (UniqueName: \"kubernetes.io/projected/b25e21c3-106e-4697-bdb2-b95a17c8a435-kube-api-access-46jx8\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:44.391746 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:44.391708    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:44.394542 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:44.394516    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:44.600799 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:44.600767    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:44.977842 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:44.977811    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"]
Nov 02 17:44:44.979066 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:44:44.979032    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25e21c3_106e_4697_bdb2_b95a17c8a435.slice/crio-661471e20084611b9e6a101d45a0a43285ae4fcdf6c00716990f6e1aaba85dfd WatchSource:0}: Error finding container 661471e20084611b9e6a101d45a0a43285ae4fcdf6c00716990f6e1aaba85dfd: Status 404 returned error can't find the container with id 661471e20084611b9e6a101d45a0a43285ae4fcdf6c00716990f6e1aaba85dfd
Nov 02 17:44:44.980524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:44.980505    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:44:45.122165 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:45.122099    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb" event={"ID":"b25e21c3-106e-4697-bdb2-b95a17c8a435","Type":"ContainerStarted","Data":"661471e20084611b9e6a101d45a0a43285ae4fcdf6c00716990f6e1aaba85dfd"}
Nov 02 17:44:46.130407 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:46.130370    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb" event={"ID":"b25e21c3-106e-4697-bdb2-b95a17c8a435","Type":"ContainerStarted","Data":"ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a"}
Nov 02 17:44:46.130880 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:46.130468    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:46.249447 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:46.249333    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb" podStartSLOduration=2.274853683 podStartE2EDuration="3.2493181s" totalImagesPullingTime="974.464417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:46.24610555 +0000 UTC m=+1316.503136173" watchObservedRunningTime="2026-11-02 17:44:46.2493181 +0000 UTC m=+1316.506348741"
Nov 02 17:44:52.143554 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:44:52.143526    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:44:59.054570 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:44:59.054533    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:45:02.507309 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:02.507257    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:02.507958 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:02.507341    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:29.055413 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:45:29.055303    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:45:57.397467 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.397433    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"]
Nov 02 17:45:57.401512 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.401492    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441079 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441048    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441090    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441246    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441282 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441278    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441392 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441361    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441426 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441391    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441426 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441421    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vsf\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-kube-api-access-z5vsf\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441526 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441506    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.441561 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.441542    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.453503 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.453475    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ceb1ae-openshift-default-dockercfg-pmptc\""
Nov 02 17:45:57.508669 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.508634    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"]
Nov 02 17:45:57.543281 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543245    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543298    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543333    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543358    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543406    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543440    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543747 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543499    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vsf\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-kube-api-access-z5vsf\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543747 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543545    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543747 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543583    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.543895 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.543742    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.544366 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.544341    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.544520 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.544495    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.545064 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.545042    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.545229 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.545208    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.547651 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.547622    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.548796 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.548767    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.601398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.601365    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vsf\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-kube-api-access-z5vsf\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.601398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.601378    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:57.715019 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:57.714984    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:58.157693 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:45:58.157667    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad8d91f_6ce0_4f10_a4fa_f67a5dd25a4e.slice/crio-022c357fb21689aea45830c570ce0c984d69fd9800cdeab35abab23e3bae0de0 WatchSource:0}: Error finding container 022c357fb21689aea45830c570ce0c984d69fd9800cdeab35abab23e3bae0de0: Status 404 returned error can't find the container with id 022c357fb21689aea45830c570ce0c984d69fd9800cdeab35abab23e3bae0de0
Nov 02 17:45:58.158103 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:58.158081    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"]
Nov 02 17:45:58.160066 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:58.160032    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:58.160173 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:58.160099    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:58.160173 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:58.160168    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:58.761814 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:58.761778    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" event={"ID":"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e","Type":"ContainerStarted","Data":"56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60"}
Nov 02 17:45:58.762277 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:58.761835    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" event={"ID":"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e","Type":"ContainerStarted","Data":"022c357fb21689aea45830c570ce0c984d69fd9800cdeab35abab23e3bae0de0"}
Nov 02 17:45:58.880997 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:58.880941    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podStartSLOduration=1.880926512 podStartE2EDuration="1.880926512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:58.877831862 +0000 UTC m=+1389.134862481" watchObservedRunningTime="2026-11-02 17:45:58.880926512 +0000 UTC m=+1389.137957213"
Nov 02 17:45:59.076388 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:45:59.076294    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:45:59.715687 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:59.715601    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:45:59.717607 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:59.717579    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 17:45:59.717748 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:45:59.717640    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 17:46:00.716056 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:00.716019    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 17:46:00.716511 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:00.716074    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 17:46:01.716248 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:01.716215    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 17:46:01.716660 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:01.716266    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 17:46:02.600876 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.600841    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"]
Nov 02 17:46:02.605059 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.605037    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.657760 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.657727    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8cbedb9f\"/\"kube-root-ca.crt\""
Nov 02 17:46:02.699239 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.699197    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8cbedb9f\"/\"default-dockercfg-8rsk4\""
Nov 02 17:46:02.707555 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.707505    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8cbedb9f\"/\"e2e-lifecycle-model-8cbedb9f-kserve-self-signed-certs\""
Nov 02 17:46:02.707555 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.707528    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8cbedb9f\"/\"openshift-service-ca.crt\""
Nov 02 17:46:02.712504 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.712470    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"]
Nov 02 17:46:02.715772 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.715740    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 17:46:02.715913 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.715820    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 17:46:02.717428 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.717399    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvdqz\" (UniqueName: \"kubernetes.io/projected/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-kube-api-access-wvdqz\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.717780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.717447    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-model-cache\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.717780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.717578    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-home\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.717780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.717676    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tls-certs\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.717780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.717717    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-dshm\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.717780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.717777    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tmp-dir\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819013 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.818972    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdqz\" (UniqueName: \"kubernetes.io/projected/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-kube-api-access-wvdqz\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819226 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819029    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-model-cache\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819226 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819103    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-home\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819226 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819191    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tls-certs\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819360 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819255    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-dshm\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819360 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819301    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tmp-dir\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819638 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819614    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-model-cache\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819725 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819663    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-home\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.819764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.819739    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tmp-dir\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.823166 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.821814    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-dshm\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.823166 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.821988    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tls-certs\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.876764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.876674    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdqz\" (UniqueName: \"kubernetes.io/projected/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-kube-api-access-wvdqz\") pod \"e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") " pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:02.921378 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:02.921344    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:03.323705 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:03.323675    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"]
Nov 02 17:46:03.324398 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:46:03.324375    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a96f5ff_16d9_4d9d_a952_b1c5eea877ed.slice/crio-6de40ad238d2f41ae72d589188e750905f421a177c3ddba7b046544571ad07ba WatchSource:0}: Error finding container 6de40ad238d2f41ae72d589188e750905f421a177c3ddba7b046544571ad07ba: Status 404 returned error can't find the container with id 6de40ad238d2f41ae72d589188e750905f421a177c3ddba7b046544571ad07ba
Nov 02 17:46:03.716024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:03.715992    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 17:46:03.716239 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:03.716049    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 17:46:03.800709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:03.800676    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq" event={"ID":"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed","Type":"ContainerStarted","Data":"84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a"}
Nov 02 17:46:03.801094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:03.800725    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq" event={"ID":"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed","Type":"ContainerStarted","Data":"6de40ad238d2f41ae72d589188e750905f421a177c3ddba7b046544571ad07ba"}
Nov 02 17:46:03.917263 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:03.917201    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq" podStartSLOduration=1.917186138 podStartE2EDuration="1.917186138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:03.915923475 +0000 UTC m=+1394.172954094" watchObservedRunningTime="2026-11-02 17:46:03.917186138 +0000 UTC m=+1394.174216758"
Nov 02 17:46:03.922339 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:03.922301    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:04.715658 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:04.715614    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 17:46:04.715854 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:04.715678    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 17:46:05.720704 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:05.720670    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:46:05.721135 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:05.721073    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:46:05.721811 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:05.721791    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:46:12.922240 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:12.922190    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:12.935776 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:12.935747    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:13.880083 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:13.879997    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:46:29.057156 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:46:29.057098    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:46:41.533160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.518170    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"]
Nov 02 17:46:41.605016 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.604981    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"]
Nov 02 17:46:41.605421 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.605399    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:41.663004 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.662964    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-215cbd\""
Nov 02 17:46:41.712155 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.712105    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-215cbd-dockercfg-4jk96\""
Nov 02 17:46:41.796203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.795285    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edbb178-4230-4105-b187-368bbcadf6e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:41.796203 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.795377    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgr6\" (UniqueName: \"kubernetes.io/projected/4edbb178-4230-4105-b187-368bbcadf6e6-kube-api-access-rmgr6\") pod \"payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:41.896338 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.896305    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgr6\" (UniqueName: \"kubernetes.io/projected/4edbb178-4230-4105-b187-368bbcadf6e6-kube-api-access-rmgr6\") pod \"payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:41.897094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.897056    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edbb178-4230-4105-b187-368bbcadf6e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:41.897933 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.897904    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edbb178-4230-4105-b187-368bbcadf6e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:41.976310 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:41.976239    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgr6\" (UniqueName: \"kubernetes.io/projected/4edbb178-4230-4105-b187-368bbcadf6e6-kube-api-access-rmgr6\") pod \"payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:42.232503 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:42.232467    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:42.714455 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:42.714423    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"]
Nov 02 17:46:42.715000 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:46:42.714972    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edbb178_4230_4105_b187_368bbcadf6e6.slice/crio-3797ebdf7534e07e9f38be68de27f6c3ab4863cc38a39d89d7b115bd8bc2e9b1 WatchSource:0}: Error finding container 3797ebdf7534e07e9f38be68de27f6c3ab4863cc38a39d89d7b115bd8bc2e9b1: Status 404 returned error can't find the container with id 3797ebdf7534e07e9f38be68de27f6c3ab4863cc38a39d89d7b115bd8bc2e9b1
Nov 02 17:46:43.091101 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:43.090999    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs" event={"ID":"4edbb178-4230-4105-b187-368bbcadf6e6","Type":"ContainerStarted","Data":"844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69"}
Nov 02 17:46:43.091284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:43.091215    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:43.091284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:43.091258    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs" event={"ID":"4edbb178-4230-4105-b187-368bbcadf6e6","Type":"ContainerStarted","Data":"3797ebdf7534e07e9f38be68de27f6c3ab4863cc38a39d89d7b115bd8bc2e9b1"}
Nov 02 17:46:43.258023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:43.257963    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs" podStartSLOduration=2.257943771 podStartE2EDuration="2.257943771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:43.207669638 +0000 UTC m=+1433.464700258" watchObservedRunningTime="2026-11-02 17:46:43.257943771 +0000 UTC m=+1433.514974393"
Nov 02 17:46:48.495722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.495687    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"]
Nov 02 17:46:48.499789 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.499772    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.551819 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.551788    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 17:46:48.551819 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.551804    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 17:46:48.551819 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.551811    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 17:46:48.552146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.551914    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gcxmh\""
Nov 02 17:46:48.591108 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.591074    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.591300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.591241    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4r9t\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-kube-api-access-q4r9t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.591300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.591276    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.591479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.591424    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.591580 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.591563    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46110b-6557-491a-84d6-eca268d92d51-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.591629 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.591616    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.591683 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.591670    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.604958 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.604926    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"]
Nov 02 17:46:48.692547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.692511    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46110b-6557-491a-84d6-eca268d92d51-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.692742 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.692581    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.692742 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.692623    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.692742 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.692665    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.692935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.692751    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9t\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-kube-api-access-q4r9t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.692935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.692788    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.692935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.692883    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.693508 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.693429    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.695432 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.695401    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.695432 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.695416    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.695555 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.695408    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46110b-6557-491a-84d6-eca268d92d51-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.695657 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.695625    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.755796 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.755722    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.756809 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.756786    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4r9t\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-kube-api-access-q4r9t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9hcr\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:48.812020 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:48.811986    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:49.246359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:49.246332    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"]
Nov 02 17:46:49.246690 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:46:49.246673    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e46110b_6557_491a_84d6_eca268d92d51.slice/crio-edcfbe2271290c8c64c295dd77520031030c322289367c642e3e23b421e96c9f WatchSource:0}: Error finding container edcfbe2271290c8c64c295dd77520031030c322289367c642e3e23b421e96c9f: Status 404 returned error can't find the container with id edcfbe2271290c8c64c295dd77520031030c322289367c642e3e23b421e96c9f
Nov 02 17:46:49.249231 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:49.249200    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:49.249317 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:49.249260    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:50.136366 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:50.136335    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr" event={"ID":"0e46110b-6557-491a-84d6-eca268d92d51","Type":"ContainerStarted","Data":"be39a54c13b4b6c23bc02fa6431beff5c384b12f221732d39ec9ee4912a32c56"}
Nov 02 17:46:50.136840 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:50.136385    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr" event={"ID":"0e46110b-6557-491a-84d6-eca268d92d51","Type":"ContainerStarted","Data":"edcfbe2271290c8c64c295dd77520031030c322289367c642e3e23b421e96c9f"}
Nov 02 17:46:50.136840 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:50.136411    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:50.138737 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:50.138710    4956 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t9hcr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:46:50.138873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:50.138764    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr" podUID="0e46110b-6557-491a-84d6-eca268d92d51" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:46:51.143757 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:51.143730    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 17:46:51.364055 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:51.363993    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr" podStartSLOduration=3.36397402 podStartE2EDuration="3.36397402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:50.349376941 +0000 UTC m=+1440.606407596" watchObservedRunningTime="2026-11-02 17:46:51.36397402 +0000 UTC m=+1441.621004640"
Nov 02 17:46:54.100604 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:46:54.100571    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:46:59.054279 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:46:59.054233    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:47:29.053613 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:47:29.053574    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:47:50.040996 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:47:50.040965    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:47:50.042112 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:47:50.042091    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:47:54.804779 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:47:54.804734    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"]
Nov 02 17:47:54.910831 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:47:54.910792    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"]
Nov 02 17:47:59.052821 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:47:59.052781    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:47:59.805483 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:47:59.805438    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerName="payload-processing" containerID="cri-o://76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b" gracePeriod=25
Nov 02 17:47:59.903369 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:47:59.903325    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:47:59.911381 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:47:59.911349    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" podUID="81858b78-1132-4786-a3ed-f93fd27130be" containerName="payload-pre-processing" containerID="cri-o://b9e35dc94d2bd1039bce0f22ade09a111c78d2598edd1653fef02e0d1ee6cb65" gracePeriod=25
Nov 02 17:48:09.902518 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:09.902480    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" podUID="81858b78-1132-4786-a3ed-f93fd27130be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:48:09.902933 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:09.902488    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:48:19.902712 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:48:19.902681    4956 logging.go:55] [core] [Channel #2064 SubChannel #2066]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.86:50052", ServerName: "10.132.0.86:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.86:50052: connect: connection refused"
Nov 02 17:48:19.905670 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:19.905615    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" podUID="81858b78-1132-4786-a3ed-f93fd27130be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:48:20.331205 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.321639    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:48:20.547332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.505816    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5344119c-53fc-4e51-8033-d3a7d4919c22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5344119c-53fc-4e51-8033-d3a7d4919c22-plugins-config-volume\") pod \"5344119c-53fc-4e51-8033-d3a7d4919c22\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") "
Nov 02 17:48:20.547332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.506015    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5344119c-53fc-4e51-8033-d3a7d4919c22-kube-api-access-ghhmf\" (UniqueName: \"kubernetes.io/projected/5344119c-53fc-4e51-8033-d3a7d4919c22-kube-api-access-ghhmf\") pod \"5344119c-53fc-4e51-8033-d3a7d4919c22\" (UID: \"5344119c-53fc-4e51-8033-d3a7d4919c22\") "
Nov 02 17:48:20.547332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.506542    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5344119c-53fc-4e51-8033-d3a7d4919c22-plugins-config-volume" pod "5344119c-53fc-4e51-8033-d3a7d4919c22" (UID: "5344119c-53fc-4e51-8033-d3a7d4919c22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:20.547332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.506666    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5344119c-53fc-4e51-8033-d3a7d4919c22-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:20.682412 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.682370    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5344119c-53fc-4e51-8033-d3a7d4919c22-kube-api-access-ghhmf" pod "5344119c-53fc-4e51-8033-d3a7d4919c22" (UID: "5344119c-53fc-4e51-8033-d3a7d4919c22"). InnerVolumeSpecName "kube-api-access-ghhmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:20.891761 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.891726    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhmf\" (UniqueName: \"kubernetes.io/projected/5344119c-53fc-4e51-8033-d3a7d4919c22-kube-api-access-ghhmf\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:20.899143 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.898282    4956 generic.go:356] "Generic (PLEG): container finished" podID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerID="76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b" exitCode=0
Nov 02 17:48:20.899335 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.899202    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.899906    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" event={"ID":"5344119c-53fc-4e51-8033-d3a7d4919c22","Type":"ContainerDied","Data":"76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b"}
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.900140    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" event={"ID":"5344119c-53fc-4e51-8033-d3a7d4919c22","Type":"ContainerDied","Data":"e9d000cfd91c767e0edde739bcd8e0dbc735964448533662d62f6fb040ea9381"}
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.900188    4956 scope.go:122] "RemoveContainer" containerID="76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b"
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.902049    4956 generic.go:356] "Generic (PLEG): container finished" podID="81858b78-1132-4786-a3ed-f93fd27130be" containerID="b9e35dc94d2bd1039bce0f22ade09a111c78d2598edd1653fef02e0d1ee6cb65" exitCode=0
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.902153    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" event={"ID":"81858b78-1132-4786-a3ed-f93fd27130be","Type":"ContainerDied","Data":"b9e35dc94d2bd1039bce0f22ade09a111c78d2598edd1653fef02e0d1ee6cb65"}
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.902195    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md" event={"ID":"81858b78-1132-4786-a3ed-f93fd27130be","Type":"ContainerDied","Data":"a11f93765c253800f46b76386f1f188760e7642dee4b916c8c32afead9848c41"}
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.902238    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a11f93765c253800f46b76386f1f188760e7642dee4b916c8c32afead9848c41"
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.902454    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.86:50052\" within 1s: context deadline exceeded"
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.902560    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"
Nov 02 17:48:20.910142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.909284    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:48:20.944650 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.940919    4956 scope.go:122] "RemoveContainer" containerID="76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b"
Nov 02 17:48:20.944650 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:48:20.942079    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b\": container with ID starting with 76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b not found: ID does not exist" containerID="76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b"
Nov 02 17:48:20.944650 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:20.942533    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b"} err="failed to get container status \"76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b\": rpc error: code = NotFound desc = could not find container \"76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b\": container with ID starting with 76b808438600afbee5901c5bb9b361d9f76065561bdb7b02ca3fe709f6fdb42b not found: ID does not exist"
Nov 02 17:48:21.102012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.101944    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"]
Nov 02 17:48:21.105320 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.103853    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81858b78-1132-4786-a3ed-f93fd27130be-kube-api-access-wqhnb\" (UniqueName: \"kubernetes.io/projected/81858b78-1132-4786-a3ed-f93fd27130be-kube-api-access-wqhnb\") pod \"81858b78-1132-4786-a3ed-f93fd27130be\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") "
Nov 02 17:48:21.105320 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.104090    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81858b78-1132-4786-a3ed-f93fd27130be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81858b78-1132-4786-a3ed-f93fd27130be-plugins-config-volume\") pod \"81858b78-1132-4786-a3ed-f93fd27130be\" (UID: \"81858b78-1132-4786-a3ed-f93fd27130be\") "
Nov 02 17:48:21.105320 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.104990    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81858b78-1132-4786-a3ed-f93fd27130be-plugins-config-volume" pod "81858b78-1132-4786-a3ed-f93fd27130be" (UID: "81858b78-1132-4786-a3ed-f93fd27130be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:21.110277 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.110241    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-jmq4q"]
Nov 02 17:48:21.113640 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.113597    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81858b78-1132-4786-a3ed-f93fd27130be-kube-api-access-wqhnb" pod "81858b78-1132-4786-a3ed-f93fd27130be" (UID: "81858b78-1132-4786-a3ed-f93fd27130be"). InnerVolumeSpecName "kube-api-access-wqhnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:21.206450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.206357    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhnb\" (UniqueName: \"kubernetes.io/projected/81858b78-1132-4786-a3ed-f93fd27130be-kube-api-access-wqhnb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:21.206450 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.206399    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81858b78-1132-4786-a3ed-f93fd27130be-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:21.902834 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.902796    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" path="/var/lib/kubelet/pods/5344119c-53fc-4e51-8033-d3a7d4919c22/volumes"
Nov 02 17:48:21.913026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:21.912994    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"
Nov 02 17:48:22.086577 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:22.086540    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"]
Nov 02 17:48:22.093030 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:22.092983    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-hs9md"]
Nov 02 17:48:23.865105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:48:23.865073    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81858b78-1132-4786-a3ed-f93fd27130be" path="/var/lib/kubelet/pods/81858b78-1132-4786-a3ed-f93fd27130be/volumes"
Nov 02 17:48:29.053613 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:48:29.053571    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:48:59.054854 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:48:59.054818    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:49:07.620257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:07.620219    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"]
Nov 02 17:49:07.620797 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:07.620448    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy" containerID="cri-o://b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac" gracePeriod=30
Nov 02 17:49:12.906090 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:12.906059    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:49:13.020869 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.020832    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-data\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.020911    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-envoy\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.020950    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438add23-9e68-4bf7-a253-84ae0bcece58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438add23-9e68-4bf7-a253-84ae0bcece58-istiod-ca-cert\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021004    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/438add23-9e68-4bf7-a253-84ae0bcece58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438add23-9e68-4bf7-a253-84ae0bcece58-istio-podinfo\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021071    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-socket\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021104    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-data" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:13.021257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021173    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-istio-token\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-istio-token\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021257 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021237    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-credential-socket\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021458 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021304    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-kube-api-access-wccz6\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-kube-api-access-wccz6\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.021458 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021360    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-certs\") pod \"438add23-9e68-4bf7-a253-84ae0bcece58\" (UID: \"438add23-9e68-4bf7-a253-84ae0bcece58\") "
Nov 02 17:49:13.022420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.021982    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-socket" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:13.022420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.022052    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-credential-socket" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:13.022420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.022018    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.022420 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.022190    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438add23-9e68-4bf7-a253-84ae0bcece58-istiod-ca-cert" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:13.022865 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.022434    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-certs" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:13.024357 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.024332    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/438add23-9e68-4bf7-a253-84ae0bcece58-istio-podinfo" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:13.024475 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.024412    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-istio-token" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:13.024573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.024556    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-envoy" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:13.024573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.024562    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-kube-api-access-wccz6" pod "438add23-9e68-4bf7-a253-84ae0bcece58" (UID: "438add23-9e68-4bf7-a253-84ae0bcece58"). InnerVolumeSpecName "kube-api-access-wccz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:13.124606 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124563    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccz6\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-kube-api-access-wccz6\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.124606 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124608    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.124803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124625    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.124803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124638    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438add23-9e68-4bf7-a253-84ae0bcece58-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.124803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124652    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438add23-9e68-4bf7-a253-84ae0bcece58-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.124803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124666    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.124803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124679    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438add23-9e68-4bf7-a253-84ae0bcece58-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.124803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.124692    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438add23-9e68-4bf7-a253-84ae0bcece58-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:13.276343 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.276310    4956 generic.go:356] "Generic (PLEG): container finished" podID="438add23-9e68-4bf7-a253-84ae0bcece58" containerID="b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac" exitCode=0
Nov 02 17:49:13.276621 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.276358    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"
Nov 02 17:49:13.276621 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.276359    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" event={"ID":"438add23-9e68-4bf7-a253-84ae0bcece58","Type":"ContainerDied","Data":"b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac"}
Nov 02 17:49:13.276621 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.276429    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8" event={"ID":"438add23-9e68-4bf7-a253-84ae0bcece58","Type":"ContainerDied","Data":"8f998fe6c520a0028b6b70268f301c67baa337ae0b85b670022b33d56ea03496"}
Nov 02 17:49:13.276621 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.276441    4956 scope.go:122] "RemoveContainer" containerID="b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac"
Nov 02 17:49:13.288242 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.288214    4956 scope.go:122] "RemoveContainer" containerID="b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac"
Nov 02 17:49:13.288614 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:13.288588    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac\": container with ID starting with b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac not found: ID does not exist" containerID="b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac"
Nov 02 17:49:13.288699 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.288629    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac"} err="failed to get container status \"b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac\": rpc error: code = NotFound desc = could not find container \"b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac\": container with ID starting with b8322e7c99a0ed5c913ac54bb8e0667458cbc46cd9f2ff1af954b2b2fc4d66ac not found: ID does not exist"
Nov 02 17:49:13.441148 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.440944    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"]
Nov 02 17:49:13.490662 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.490629    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-t96v8"]
Nov 02 17:49:13.865933 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:13.865897    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" path="/var/lib/kubelet/pods/438add23-9e68-4bf7-a253-84ae0bcece58/volumes"
Nov 02 17:49:14.338328 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.338296    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"]
Nov 02 17:49:14.338758 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.338511    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb" podUID="b25e21c3-106e-4697-bdb2-b95a17c8a435" containerName="maas-api" containerID="cri-o://ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a" gracePeriod=30
Nov 02 17:49:14.631507 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.631473    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:49:14.651737 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.651636    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25e21c3-106e-4697-bdb2-b95a17c8a435-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21c3-106e-4697-bdb2-b95a17c8a435-openshift-service-ca\") pod \"b25e21c3-106e-4697-bdb2-b95a17c8a435\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") "
Nov 02 17:49:14.652437 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.652405    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25e21c3-106e-4697-bdb2-b95a17c8a435-openshift-service-ca" pod "b25e21c3-106e-4697-bdb2-b95a17c8a435" (UID: "b25e21c3-106e-4697-bdb2-b95a17c8a435"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:14.752847 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.752809    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-metrics-certs\") pod \"b25e21c3-106e-4697-bdb2-b95a17c8a435\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") "
Nov 02 17:49:14.753048 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.752908    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25e21c3-106e-4697-bdb2-b95a17c8a435-kube-api-access-46jx8\" (UniqueName: \"kubernetes.io/projected/b25e21c3-106e-4697-bdb2-b95a17c8a435-kube-api-access-46jx8\") pod \"b25e21c3-106e-4697-bdb2-b95a17c8a435\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") "
Nov 02 17:49:14.753048 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.752960    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls\") pod \"b25e21c3-106e-4697-bdb2-b95a17c8a435\" (UID: \"b25e21c3-106e-4697-bdb2-b95a17c8a435\") "
Nov 02 17:49:14.753508 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.753487    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21c3-106e-4697-bdb2-b95a17c8a435-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:14.756349 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.756324    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls" pod "b25e21c3-106e-4697-bdb2-b95a17c8a435" (UID: "b25e21c3-106e-4697-bdb2-b95a17c8a435"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:49:14.756451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.756365    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25e21c3-106e-4697-bdb2-b95a17c8a435-kube-api-access-46jx8" pod "b25e21c3-106e-4697-bdb2-b95a17c8a435" (UID: "b25e21c3-106e-4697-bdb2-b95a17c8a435"). InnerVolumeSpecName "kube-api-access-46jx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:14.758285 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.758247    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-metrics-certs" pod "b25e21c3-106e-4697-bdb2-b95a17c8a435" (UID: "b25e21c3-106e-4697-bdb2-b95a17c8a435"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:49:14.854777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.854685    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:14.854777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.854718    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46jx8\" (UniqueName: \"kubernetes.io/projected/b25e21c3-106e-4697-bdb2-b95a17c8a435-kube-api-access-46jx8\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:14.854777 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:14.854729    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21c3-106e-4697-bdb2-b95a17c8a435-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:15.305145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.304043    4956 generic.go:356] "Generic (PLEG): container finished" podID="b25e21c3-106e-4697-bdb2-b95a17c8a435" containerID="ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a" exitCode=0
Nov 02 17:49:15.305145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.304149    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb" event={"ID":"b25e21c3-106e-4697-bdb2-b95a17c8a435","Type":"ContainerDied","Data":"ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a"}
Nov 02 17:49:15.305145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.304197    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb" event={"ID":"b25e21c3-106e-4697-bdb2-b95a17c8a435","Type":"ContainerDied","Data":"661471e20084611b9e6a101d45a0a43285ae4fcdf6c00716990f6e1aaba85dfd"}
Nov 02 17:49:15.305145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.304222    4956 scope.go:122] "RemoveContainer" containerID="ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a"
Nov 02 17:49:15.305145 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.304364    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"
Nov 02 17:49:15.345945 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.345366    4956 scope.go:122] "RemoveContainer" containerID="ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a"
Nov 02 17:49:15.346691 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:15.346555    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a\": container with ID starting with ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a not found: ID does not exist" containerID="ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a"
Nov 02 17:49:15.346691 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.346596    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a"} err="failed to get container status \"ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a\": rpc error: code = NotFound desc = could not find container \"ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a\": container with ID starting with ee8e4e395a3ef04ee5b15b3b45c1c00fd0ec6accb927961fa896b0f6fb2c0d6a not found: ID does not exist"
Nov 02 17:49:15.486363 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.486325    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"]
Nov 02 17:49:15.536076 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.536045    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-4fvcb"]
Nov 02 17:49:15.866188 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.866154    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25e21c3-106e-4697-bdb2-b95a17c8a435" path="/var/lib/kubelet/pods/b25e21c3-106e-4697-bdb2-b95a17c8a435/volumes"
Nov 02 17:49:15.980540 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.980509    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"]
Nov 02 17:49:15.981093 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981079    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerName="payload-processing"
Nov 02 17:49:15.981151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981094    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerName="payload-processing"
Nov 02 17:49:15.981151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981105    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b25e21c3-106e-4697-bdb2-b95a17c8a435" containerName="maas-api"
Nov 02 17:49:15.981151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981110    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25e21c3-106e-4697-bdb2-b95a17c8a435" containerName="maas-api"
Nov 02 17:49:15.981151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981139    4956 cpu_manager.go:414] "containerMap: removing container" podUID="81858b78-1132-4786-a3ed-f93fd27130be" containerName="payload-pre-processing"
Nov 02 17:49:15.981151 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981145    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81858b78-1132-4786-a3ed-f93fd27130be" containerName="payload-pre-processing"
Nov 02 17:49:15.981307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981163    4956 cpu_manager.go:414] "containerMap: removing container" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy"
Nov 02 17:49:15.981307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981169    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy"
Nov 02 17:49:15.981307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981258    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="438add23-9e68-4bf7-a253-84ae0bcece58" containerName="istio-proxy"
Nov 02 17:49:15.981307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981270    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5344119c-53fc-4e51-8033-d3a7d4919c22" containerName="payload-processing"
Nov 02 17:49:15.981307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981276    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25e21c3-106e-4697-bdb2-b95a17c8a435" containerName="maas-api"
Nov 02 17:49:15.981307 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.981285    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="81858b78-1132-4786-a3ed-f93fd27130be" containerName="payload-pre-processing"
Nov 02 17:49:15.985977 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:15.985960    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.038218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.037950    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f0f3e1-serving-cert\""
Nov 02 17:49:16.041995 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.041968    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"]
Nov 02 17:49:16.068301 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.068265    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.068479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.068384    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-openshift-service-ca\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.068479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.068466    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-metrics-certs\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.068571 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.068539    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9b2\" (UniqueName: \"kubernetes.io/projected/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-kube-api-access-cp9b2\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.170333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.170228    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.170333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.170308    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-openshift-service-ca\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.170554 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.170383    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-metrics-certs\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.170554 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:16.170419    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-serving-cert: secret "maas-api-e2e-api-b-f0f3e1-serving-cert" not found
Nov 02 17:49:16.170554 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.170428    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9b2\" (UniqueName: \"kubernetes.io/projected/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-kube-api-access-cp9b2\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.170554 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:16.170490    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls podName:d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43 nodeName:}" failed. No retries permitted until 2026-11-02 17:49:16.67046987 +0000 UTC m=+1586.927500482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls") pod "maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw" (UID: "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43") : secret "maas-api-e2e-api-b-f0f3e1-serving-cert" not found
Nov 02 17:49:16.171284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.171253    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-openshift-service-ca\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.173342 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.173319    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-metrics-certs\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.227814 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.227775    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9b2\" (UniqueName: \"kubernetes.io/projected/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-kube-api-access-cp9b2\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.676393 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.676349    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.679040 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.679019    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls\") pod \"maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:16.899551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:16.899515    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:17.280564 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:17.280536    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"]
Nov 02 17:49:17.282865 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:49:17.282837    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b03a06_5b8f_49ec_bcd5_4a5eb6a2ed43.slice/crio-3940c194b1ecb2ab8d5c9253eabb83151b72b88592b070be320a9344b5790853 WatchSource:0}: Error finding container 3940c194b1ecb2ab8d5c9253eabb83151b72b88592b070be320a9344b5790853: Status 404 returned error can't find the container with id 3940c194b1ecb2ab8d5c9253eabb83151b72b88592b070be320a9344b5790853
Nov 02 17:49:17.318754 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:17.318718    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw" event={"ID":"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43","Type":"ContainerStarted","Data":"3940c194b1ecb2ab8d5c9253eabb83151b72b88592b070be320a9344b5790853"}
Nov 02 17:49:18.327042 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:18.327001    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw" event={"ID":"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43","Type":"ContainerStarted","Data":"1ece4095335f9b0779ca04b78e88224c86b88bc1e93bca094ab554e5001ad206"}
Nov 02 17:49:18.327452 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:18.327084    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:18.445673 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:18.445567    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw" podStartSLOduration=2.579915627 podStartE2EDuration="3.445550001s" totalImagesPullingTime="865.634374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:18.44384056 +0000 UTC m=+1588.700871180" watchObservedRunningTime="2026-11-02 17:49:18.445550001 +0000 UTC m=+1588.702580657"
Nov 02 17:49:24.340172 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:24.340139    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:49:26.742244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.742202    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"]
Nov 02 17:49:26.749355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.749317    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.801016 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.800975    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"]
Nov 02 17:49:26.814216 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.814173    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6qg\" (UniqueName: \"kubernetes.io/projected/c447e80a-d759-4cc7-8913-9e101b649d1e-kube-api-access-lb6qg\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.814216 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.814218    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.814489 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.814321    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.814489 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.814396    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c447e80a-d759-4cc7-8913-9e101b649d1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.834697 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.834658    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3fa615-serving-cert\""
Nov 02 17:49:26.915773 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.915739    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6qg\" (UniqueName: \"kubernetes.io/projected/c447e80a-d759-4cc7-8913-9e101b649d1e-kube-api-access-lb6qg\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.915773 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.915783    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.916057 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.915829    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.916057 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.915856    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c447e80a-d759-4cc7-8913-9e101b649d1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.916057 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:26.915957    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-serving-cert: secret "maas-api-e2e-ipp-a-3fa615-serving-cert" not found
Nov 02 17:49:26.916057 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:26.916055    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls podName:c447e80a-d759-4cc7-8913-9e101b649d1e nodeName:}" failed. No retries permitted until 2026-11-02 17:49:27.416021223 +0000 UTC m=+1597.673051835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls") pod "maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864" (UID: "c447e80a-d759-4cc7-8913-9e101b649d1e") : secret "maas-api-e2e-ipp-a-3fa615-serving-cert" not found
Nov 02 17:49:26.916812 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.916785    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c447e80a-d759-4cc7-8913-9e101b649d1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.919110 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.919088    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:26.972684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:26.972644    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6qg\" (UniqueName: \"kubernetes.io/projected/c447e80a-d759-4cc7-8913-9e101b649d1e-kube-api-access-lb6qg\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:27.424265 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:27.424227    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:27.426991 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:27.426962    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:27.665354 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:27.665315    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:28.049314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:28.049284    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"]
Nov 02 17:49:28.060437 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:49:28.056797    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc447e80a_d759_4cc7_8913_9e101b649d1e.slice/crio-9fd80801d44200f96fdec5d00f6e5f3c3e267317025b3ed71f7bd80bc7a128e5 WatchSource:0}: Error finding container 9fd80801d44200f96fdec5d00f6e5f3c3e267317025b3ed71f7bd80bc7a128e5: Status 404 returned error can't find the container with id 9fd80801d44200f96fdec5d00f6e5f3c3e267317025b3ed71f7bd80bc7a128e5
Nov 02 17:49:28.395945 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:28.395838    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864" event={"ID":"c447e80a-d759-4cc7-8913-9e101b649d1e","Type":"ContainerStarted","Data":"9fd80801d44200f96fdec5d00f6e5f3c3e267317025b3ed71f7bd80bc7a128e5"}
Nov 02 17:49:29.058985 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:29.058951    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:49:29.403838 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:29.403803    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864" event={"ID":"c447e80a-d759-4cc7-8913-9e101b649d1e","Type":"ContainerStarted","Data":"85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485"}
Nov 02 17:49:29.404010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:29.403877    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:29.520135 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:29.520065    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864" podStartSLOduration=2.706094179 podStartE2EDuration="3.520050642s" totalImagesPullingTime="813.956463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:29.51647066 +0000 UTC m=+1599.773501279" watchObservedRunningTime="2026-11-02 17:49:29.520050642 +0000 UTC m=+1599.777081263"
Nov 02 17:49:35.416263 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:35.416235    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:49:46.028261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:46.028224    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"]
Nov 02 17:49:46.028815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:46.028433    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" containerID="cri-o://8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee" gracePeriod=30
Nov 02 17:49:50.617867 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:50.617825    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:49:50.618356 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:50.617899    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:49:51.374497 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.374470    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:49:51.495268 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495235    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-socket\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495454 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495326    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-podinfo\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495454 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495366    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-certs\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495542 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495510    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-credential-socket\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495596 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495548    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-kube-api-access-tw669\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-kube-api-access-tw669\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495596 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495577    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-token\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-token\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495596 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495572    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-socket" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:51.495759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495625    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3560d50a-7609-4c5b-99f8-d094f0cccc50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3560d50a-7609-4c5b-99f8-d094f0cccc50-istiod-ca-cert\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495672    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-certs" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:51.495759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495738    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-data\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.495759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495748    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-credential-socket" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:51.495951 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495782    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-envoy\") pod \"3560d50a-7609-4c5b-99f8-d094f0cccc50\" (UID: \"3560d50a-7609-4c5b-99f8-d094f0cccc50\") "
Nov 02 17:49:51.496011 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.495990    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-data" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:51.496103 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.496074    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3560d50a-7609-4c5b-99f8-d094f0cccc50-istiod-ca-cert" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:51.496723 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.496696    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.496851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.496726    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.496851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.496742    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.496851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.496757    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3560d50a-7609-4c5b-99f8-d094f0cccc50-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.496851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.496772    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.498073 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.498044    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-kube-api-access-tw669" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "kube-api-access-tw669". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:51.498403 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.498377    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-envoy" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:51.498518 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.498473    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-token" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:51.498518 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.498497    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-podinfo" pod "3560d50a-7609-4c5b-99f8-d094f0cccc50" (UID: "3560d50a-7609-4c5b-99f8-d094f0cccc50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:51.551994 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.551951    4956 generic.go:356] "Generic (PLEG): container finished" podID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerID="8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee" exitCode=0
Nov 02 17:49:51.552169 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.552013    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"
Nov 02 17:49:51.552169 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.552011    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" event={"ID":"3560d50a-7609-4c5b-99f8-d094f0cccc50","Type":"ContainerDied","Data":"8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee"}
Nov 02 17:49:51.552263 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.552171    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6" event={"ID":"3560d50a-7609-4c5b-99f8-d094f0cccc50","Type":"ContainerDied","Data":"9c1d1dc2c1199c5968bec0ff907a88ebe42e35b5f94c23e32ed5f1e037da36e9"}
Nov 02 17:49:51.552263 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.552199    4956 scope.go:122] "RemoveContainer" containerID="8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee"
Nov 02 17:49:51.591367 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.573198    4956 scope.go:122] "RemoveContainer" containerID="8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee"
Nov 02 17:49:51.591367 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:51.573759    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee\": container with ID starting with 8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee not found: ID does not exist" containerID="8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee"
Nov 02 17:49:51.591367 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.573797    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee"} err="failed to get container status \"8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee\": rpc error: code = NotFound desc = could not find container \"8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee\": container with ID starting with 8b634a69b51892791894193904632b87bfb613fc172bd87516fe200f5b690fee not found: ID does not exist"
Nov 02 17:49:51.597860 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.597773    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.597860 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.597808    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.597860 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.597824    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw669\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-kube-api-access-tw669\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.597860 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.597849    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3560d50a-7609-4c5b-99f8-d094f0cccc50-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:51.721071 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.721034    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"]
Nov 02 17:49:51.727370 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.727334    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-86lh6"]
Nov 02 17:49:51.865047 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:49:51.864951    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" path="/var/lib/kubelet/pods/3560d50a-7609-4c5b-99f8-d094f0cccc50/volumes"
Nov 02 17:49:59.053562 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:49:59.053524    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:50:29.053641 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:50:29.053595    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:50:32.023397 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:32.023307    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"]
Nov 02 17:50:37.024026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.023980    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs" podUID="4edbb178-4230-4105-b187-368bbcadf6e6" containerName="payload-processing" containerID="cri-o://844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69" gracePeriod=25
Nov 02 17:50:37.311716 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.311691    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:50:37.331845 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.331820    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4edbb178-4230-4105-b187-368bbcadf6e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edbb178-4230-4105-b187-368bbcadf6e6-plugins-config-volume\") pod \"4edbb178-4230-4105-b187-368bbcadf6e6\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") "
Nov 02 17:50:37.331968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.331934    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edbb178-4230-4105-b187-368bbcadf6e6-kube-api-access-rmgr6\" (UniqueName: \"kubernetes.io/projected/4edbb178-4230-4105-b187-368bbcadf6e6-kube-api-access-rmgr6\") pod \"4edbb178-4230-4105-b187-368bbcadf6e6\" (UID: \"4edbb178-4230-4105-b187-368bbcadf6e6\") "
Nov 02 17:50:37.332274 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.332250    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edbb178-4230-4105-b187-368bbcadf6e6-plugins-config-volume" pod "4edbb178-4230-4105-b187-368bbcadf6e6" (UID: "4edbb178-4230-4105-b187-368bbcadf6e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:37.332570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.332553    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edbb178-4230-4105-b187-368bbcadf6e6-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:37.334294 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.334268    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edbb178-4230-4105-b187-368bbcadf6e6-kube-api-access-rmgr6" pod "4edbb178-4230-4105-b187-368bbcadf6e6" (UID: "4edbb178-4230-4105-b187-368bbcadf6e6"). InnerVolumeSpecName "kube-api-access-rmgr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:37.433737 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.433703    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgr6\" (UniqueName: \"kubernetes.io/projected/4edbb178-4230-4105-b187-368bbcadf6e6-kube-api-access-rmgr6\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:37.902357 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.902317    4956 generic.go:356] "Generic (PLEG): container finished" podID="4edbb178-4230-4105-b187-368bbcadf6e6" containerID="844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69" exitCode=0
Nov 02 17:50:37.902559 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.902382    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"
Nov 02 17:50:37.902559 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.902400    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs" event={"ID":"4edbb178-4230-4105-b187-368bbcadf6e6","Type":"ContainerDied","Data":"844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69"}
Nov 02 17:50:37.902559 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.902492    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs" event={"ID":"4edbb178-4230-4105-b187-368bbcadf6e6","Type":"ContainerDied","Data":"3797ebdf7534e07e9f38be68de27f6c3ab4863cc38a39d89d7b115bd8bc2e9b1"}
Nov 02 17:50:37.902559 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.902506    4956 scope.go:122] "RemoveContainer" containerID="844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69"
Nov 02 17:50:37.912655 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.912634    4956 scope.go:122] "RemoveContainer" containerID="844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69"
Nov 02 17:50:37.912999 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:50:37.912977    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69\": container with ID starting with 844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69 not found: ID does not exist" containerID="844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69"
Nov 02 17:50:37.913086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:37.913008    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69"} err="failed to get container status \"844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69\": rpc error: code = NotFound desc = could not find container \"844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69\": container with ID starting with 844a883c8815b9bfbc7ec912c927a0d008aad875fb5a100b7cbd11057e4ecc69 not found: ID does not exist"
Nov 02 17:50:38.069732 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:38.069692    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"]
Nov 02 17:50:38.119081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:38.119051    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-215cbd-ff68b5c45-wh2qs"]
Nov 02 17:50:39.865662 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:39.865629    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4edbb178-4230-4105-b187-368bbcadf6e6" path="/var/lib/kubelet/pods/4edbb178-4230-4105-b187-368bbcadf6e6/volumes"
Nov 02 17:50:50.169967 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:50.169936    4956 scope.go:122] "RemoveContainer" containerID="b9e35dc94d2bd1039bce0f22ade09a111c78d2598edd1653fef02e0d1ee6cb65"
Nov 02 17:50:55.599060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.599014    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"]
Nov 02 17:50:55.599870 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.599845    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy"
Nov 02 17:50:55.599982 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.599872    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy"
Nov 02 17:50:55.599982 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.599920    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4edbb178-4230-4105-b187-368bbcadf6e6" containerName="payload-processing"
Nov 02 17:50:55.599982 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.599929    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edbb178-4230-4105-b187-368bbcadf6e6" containerName="payload-processing"
Nov 02 17:50:55.600178 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.600164    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edbb178-4230-4105-b187-368bbcadf6e6" containerName="payload-processing"
Nov 02 17:50:55.600243 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.600191    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3560d50a-7609-4c5b-99f8-d094f0cccc50" containerName="istio-proxy"
Nov 02 17:50:55.605219 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.605195    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.657449 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.657414    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8cbedb9f-openshift-default-dockercfg-f6kqf\""
Nov 02 17:50:55.774381 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774278    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d3b545-2d27-417c-b815-d56360a399ed-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.774381 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774376    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.774381 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774407    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.774381 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774433    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d3b545-2d27-417c-b815-d56360a399ed-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.774828 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774514    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.774828 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774651    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.774828 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774785    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6nz\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-kube-api-access-4s6nz\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.774828 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774824    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.775056 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.774885    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.820081 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.820042    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"]
Nov 02 17:50:55.876131 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876023    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d3b545-2d27-417c-b815-d56360a399ed-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876312 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876160    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876312 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876200    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876312 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876225    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d3b545-2d27-417c-b815-d56360a399ed-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876486 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876353    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876486 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876401    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876593 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876516    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6nz\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-kube-api-access-4s6nz\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876593 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876548    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876593 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876583    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876740 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876608    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876740 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876660    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876831    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d3b545-2d27-417c-b815-d56360a399ed-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.876905 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.876852    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.877107 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.877049    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.878793 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.878771    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.879013 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.878997    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d3b545-2d27-417c-b815-d56360a399ed-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.935664 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.935626    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:55.936956 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:55.936931    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6nz\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-kube-api-access-4s6nz\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:56.218668 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:56.218637    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:56.691356 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:50:56.691328    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d3b545_2d27_417c_b815_d56360a399ed.slice/crio-5c1a49c9086dc11f518c703dceae745082767e6104d2e69f2f22a476d14a11b2 WatchSource:0}: Error finding container 5c1a49c9086dc11f518c703dceae745082767e6104d2e69f2f22a476d14a11b2: Status 404 returned error can't find the container with id 5c1a49c9086dc11f518c703dceae745082767e6104d2e69f2f22a476d14a11b2
Nov 02 17:50:56.692424 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:56.692402    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"]
Nov 02 17:50:56.693865 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:56.693835    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:56.693935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:56.693897    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:56.693935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:56.693925    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:57.025511 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:57.025471    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" event={"ID":"72d3b545-2d27-417c-b815-d56360a399ed","Type":"ContainerStarted","Data":"e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08"}
Nov 02 17:50:57.025709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:57.025539    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" event={"ID":"72d3b545-2d27-417c-b815-d56360a399ed","Type":"ContainerStarted","Data":"5c1a49c9086dc11f518c703dceae745082767e6104d2e69f2f22a476d14a11b2"}
Nov 02 17:50:57.149046 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:57.148968    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podStartSLOduration=2.148950628 podStartE2EDuration="2.148950628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:57.144634098 +0000 UTC m=+1687.401664709" watchObservedRunningTime="2026-11-02 17:50:57.148950628 +0000 UTC m=+1687.405981248"
Nov 02 17:50:57.218905 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:57.218869    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:50:57.220543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:57.220517    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 17:50:57.220685 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:57.220573    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 17:50:58.219287 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:58.219251    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 17:50:58.219733 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:58.219302    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 17:50:59.077984 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:50:59.077930    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:50:59.219302 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:59.219261    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 17:50:59.219781 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:50:59.219326    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 17:51:00.222185 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:00.221507    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 17:51:00.222185 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:00.221581    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 17:51:01.219077 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:01.219041    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 17:51:01.219298 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:01.219097    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 17:51:02.219215 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:02.219176    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 17:51:02.219719 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:02.219253    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 17:51:03.254250 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:03.254217    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:51:03.255012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:03.254983    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:51:03.256155 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:03.256089    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:51:10.801235 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.801189    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"]
Nov 02 17:51:10.806621 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.806593    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916065 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916025    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916076    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ccl\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-kube-api-access-v9ccl\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916256    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916379 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916299    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916441 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916424    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916562 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916497    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916562 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916555    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916695    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:10.916785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:10.916772    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018298 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018249    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018497 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018331    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018497 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018380    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018497 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018408    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ccl\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-kube-api-access-v9ccl\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018497 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018452    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018697 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018636    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018758 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018731    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018776    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018792    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018847    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018886    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.018918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.018904    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.019291 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.019269    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.019487 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.019466    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.021019 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.020996    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.021161 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.021145    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.049905 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.049869    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"]
Nov 02 17:51:11.081190 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.081080    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.084251 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.083185    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ccl\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-kube-api-access-v9ccl\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.124371 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.124336    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:11.568952 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.568915    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"]
Nov 02 17:51:11.595454 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:51:11.595426    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b33a1ff_31e0_4a38_8036_9a46ed1fdcac.slice/crio-632dbe4e16eb0017c5add94ae41d30e9585a33646c43235697e3922fa7bb7bc7 WatchSource:0}: Error finding container 632dbe4e16eb0017c5add94ae41d30e9585a33646c43235697e3922fa7bb7bc7: Status 404 returned error can't find the container with id 632dbe4e16eb0017c5add94ae41d30e9585a33646c43235697e3922fa7bb7bc7
Nov 02 17:51:11.598013 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.597977    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:11.598105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.598048    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:11.598105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:11.598076    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:12.187473 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:12.187426    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" event={"ID":"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac","Type":"ContainerStarted","Data":"4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9"}
Nov 02 17:51:12.188054 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:12.187537    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" event={"ID":"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac","Type":"ContainerStarted","Data":"632dbe4e16eb0017c5add94ae41d30e9585a33646c43235697e3922fa7bb7bc7"}
Nov 02 17:51:12.354353 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:12.354294    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" podStartSLOduration=2.354278887 podStartE2EDuration="2.354278887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:12.351415323 +0000 UTC m=+1702.608445944" watchObservedRunningTime="2026-11-02 17:51:12.354278887 +0000 UTC m=+1702.611309506"
Nov 02 17:51:13.124931 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:13.124888    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:13.150189 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:13.142281    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:51:13.150189 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:13.142368    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:51:14.144147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:14.134597    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:51:14.144147 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:14.134663    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:51:15.125238 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:15.125201    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:51:15.125553 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:15.125515    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:51:16.150358 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:16.150315    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:51:16.150801 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:16.150407    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:51:17.138146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:17.134245    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:51:17.138146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:17.134341    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:51:18.200571 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:18.200538    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:18.201547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:18.201522    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:18.204356 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:18.204331    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:51:22.902574 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:22.902531    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"]
Nov 02 17:51:22.903089 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:22.902778    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw" podUID="d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" containerName="maas-api" containerID="cri-o://1ece4095335f9b0779ca04b78e88224c86b88bc1e93bca094ab554e5001ad206" gracePeriod=30
Nov 02 17:51:23.310026 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.309996    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw" event={"ID":"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43","Type":"ContainerDied","Data":"1ece4095335f9b0779ca04b78e88224c86b88bc1e93bca094ab554e5001ad206"}
Nov 02 17:51:23.310206 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.310135    4956 generic.go:356] "Generic (PLEG): container finished" podID="d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" containerID="1ece4095335f9b0779ca04b78e88224c86b88bc1e93bca094ab554e5001ad206" exitCode=0
Nov 02 17:51:23.310250 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.310223    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw" event={"ID":"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43","Type":"ContainerDied","Data":"3940c194b1ecb2ab8d5c9253eabb83151b72b88592b070be320a9344b5790853"}
Nov 02 17:51:23.310293 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.310249    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3940c194b1ecb2ab8d5c9253eabb83151b72b88592b070be320a9344b5790853"
Nov 02 17:51:23.313330 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.313302    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:51:23.444146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.443420    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-metrics-certs\") pod \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") "
Nov 02 17:51:23.444146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.443518    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls\") pod \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") "
Nov 02 17:51:23.444146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.443624    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-kube-api-access-cp9b2\" (UniqueName: \"kubernetes.io/projected/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-kube-api-access-cp9b2\") pod \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") "
Nov 02 17:51:23.444146 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.443677    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-openshift-service-ca\") pod \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\" (UID: \"d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43\") "
Nov 02 17:51:23.468540 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.468503    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-kube-api-access-cp9b2" pod "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" (UID: "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43"). InnerVolumeSpecName "kube-api-access-cp9b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:23.472653 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.472617    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-openshift-service-ca" pod "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" (UID: "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:23.473480 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.473438    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-metrics-certs" pod "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" (UID: "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:23.515460 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.509280    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls" pod "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" (UID: "d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:23.561707 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.561669    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9b2\" (UniqueName: \"kubernetes.io/projected/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-kube-api-access-cp9b2\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:23.561931 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.561915    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:23.562022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.562012    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:23.562110 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:23.562100    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:24.317350 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:24.317302    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"
Nov 02 17:51:24.483481 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:24.483445    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"]
Nov 02 17:51:24.532898 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:24.532869    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f0f3e1-cbc85779b-bbgqw"]
Nov 02 17:51:25.865686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.865651    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" path="/var/lib/kubelet/pods/d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43/volumes"
Nov 02 17:51:25.896371 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.896341    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"]
Nov 02 17:51:25.896984 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.896969    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" containerName="maas-api"
Nov 02 17:51:25.897080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.896986    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" containerName="maas-api"
Nov 02 17:51:25.897155 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.897132    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b03a06-5b8f-49ec-bcd5-4a5eb6a2ed43" containerName="maas-api"
Nov 02 17:51:25.901737 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.901717    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.995588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.995551    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.995735 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.995600    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afec1af-daec-419c-bdae-c6ec7e8d6452-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.995735 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.995696    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.995834 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.995815    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.995868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.995858    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.995928 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.995912    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9b7\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-kube-api-access-rp9b7\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.996060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.996045    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.996098 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.996079    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:25.996194 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:25.996182    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.005677 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.005644    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"]
Nov 02 17:51:26.097844 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.097812    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098038 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.097853    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098038 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.097896    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098038 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.097928    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098038 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098024    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afec1af-daec-419c-bdae-c6ec7e8d6452-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098107    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098235    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098284    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098340    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098554 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098508    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098554 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098345    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9b7\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-kube-api-access-rp9b7\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098649 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098622    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.098764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.098746    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.099113 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.099087    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afec1af-daec-419c-bdae-c6ec7e8d6452-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.100569 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.100548    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.100720 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.100704    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.158804 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.158724    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9b7\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-kube-api-access-rp9b7\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.160284 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.160257    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.216160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.216103    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:26.673831 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.673801    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"]
Nov 02 17:51:26.674948 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:51:26.674924    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afec1af_daec_419c_bdae_c6ec7e8d6452.slice/crio-2d6c21e2cd81cdb579409d1712a49f8613b4de2dd73f96a87611e41b026ef732 WatchSource:0}: Error finding container 2d6c21e2cd81cdb579409d1712a49f8613b4de2dd73f96a87611e41b026ef732: Status 404 returned error can't find the container with id 2d6c21e2cd81cdb579409d1712a49f8613b4de2dd73f96a87611e41b026ef732
Nov 02 17:51:26.677593 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.677559    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:26.677701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.677639    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:26.677701 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:26.677684    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:27.357686 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:27.357648    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" event={"ID":"9afec1af-daec-419c-bdae-c6ec7e8d6452","Type":"ContainerStarted","Data":"a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e"}
Nov 02 17:51:27.358201 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:27.357734    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" event={"ID":"9afec1af-daec-419c-bdae-c6ec7e8d6452","Type":"ContainerStarted","Data":"2d6c21e2cd81cdb579409d1712a49f8613b4de2dd73f96a87611e41b026ef732"}
Nov 02 17:51:27.563550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:27.563500    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" podStartSLOduration=2.563485749 podStartE2EDuration="2.563485749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:27.524500288 +0000 UTC m=+1717.781530988" watchObservedRunningTime="2026-11-02 17:51:27.563485749 +0000 UTC m=+1717.820516431"
Nov 02 17:51:28.217321 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:28.217256    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:28.218762 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:28.218739    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 17:51:28.218834 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:28.218789    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 17:51:29.076479 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:51:29.076445    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:51:29.217388 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:29.217350    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 17:51:29.217586 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:29.217420    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 17:51:30.216407 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:30.216378    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 17:51:30.216856 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:30.216434    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 17:51:31.216890 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:31.216855    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 17:51:31.217423 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:31.216930    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 17:51:32.216974 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:32.216941    4956 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 17:51:32.217525 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:32.216996    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 17:51:33.232110 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:33.232077    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:33.233714 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:33.233688    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:33.234478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:33.234456    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:51:38.051837 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:38.051797    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"]
Nov 02 17:51:48.766344 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.766312    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"]
Nov 02 17:51:48.770179 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.770161    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.822727 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.822696    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6f23bd-openshift-default-dockercfg-9wmkr\""
Nov 02 17:51:48.879500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.879447    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.879703 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.879539    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.879703 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.879650    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.879823 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.879768    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.879892 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.879875    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.879950 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.879906    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhf27\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-kube-api-access-qhf27\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.880006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.879953    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.880063 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.880034    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.880224 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.880208    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981068 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981034    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981091    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981154    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981261 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981177    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981370 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981303    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981411 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981369    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981411 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981394    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhf27\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-kube-api-access-qhf27\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981509 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981446    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981509 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981483    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981509 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981491    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981542    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981603    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981814    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.981991 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.981968    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.983912 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.983890    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.984134 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.984104    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:48.996111 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:48.996072    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"]
Nov 02 17:51:49.039097 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:49.039027    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:49.039337 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:49.039311    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhf27\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-kube-api-access-qhf27\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:49.091107 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:49.091071    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:49.526915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:49.526885    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"]
Nov 02 17:51:49.560059 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:51:49.560029    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d080ee_bf15_4119_83f0_b3cfc14a3c73.slice/crio-27d6f8960a5052ed52d36ffe5a3c420ea3b50e559ea396767e12c5025c7beae7 WatchSource:0}: Error finding container 27d6f8960a5052ed52d36ffe5a3c420ea3b50e559ea396767e12c5025c7beae7: Status 404 returned error can't find the container with id 27d6f8960a5052ed52d36ffe5a3c420ea3b50e559ea396767e12c5025c7beae7
Nov 02 17:51:49.562785 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:49.562754    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:49.562885 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:49.562821    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:49.562885 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:49.562850    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:50.519969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:50.519909    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" event={"ID":"f4d080ee-bf15-4119-83f0-b3cfc14a3c73","Type":"ContainerStarted","Data":"2500cb3a4027b0dba50689d522dcac578b63209bc29de3661fc28cd389b850e1"}
Nov 02 17:51:50.519969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:50.519974    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" event={"ID":"f4d080ee-bf15-4119-83f0-b3cfc14a3c73","Type":"ContainerStarted","Data":"27d6f8960a5052ed52d36ffe5a3c420ea3b50e559ea396767e12c5025c7beae7"}
Nov 02 17:51:50.765648 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:50.765597    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" podStartSLOduration=2.765582321 podStartE2EDuration="2.765582321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:50.713434067 +0000 UTC m=+1740.970464687" watchObservedRunningTime="2026-11-02 17:51:50.765582321 +0000 UTC m=+1741.022612943"
Nov 02 17:51:51.092001 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:51.091972    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:51.093319 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:51.093293    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 17:51:51.093446 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:51.093352    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 17:51:52.091530 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:52.091487    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 17:51:52.091943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:52.091547    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 17:51:53.073547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.073489    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq" podUID="5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" containerName="main" containerID="cri-o://84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a" gracePeriod=45
Nov 02 17:51:53.092295 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.092266    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 17:51:53.092681 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.092325    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 17:51:53.398547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.398517    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:51:53.568750 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.568711    4956 generic.go:356] "Generic (PLEG): container finished" podID="5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" containerID="84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a" exitCode=0
Nov 02 17:51:53.568938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.568785    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq" event={"ID":"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed","Type":"ContainerDied","Data":"84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a"}
Nov 02 17:51:53.568938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.568825    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq" event={"ID":"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed","Type":"ContainerDied","Data":"6de40ad238d2f41ae72d589188e750905f421a177c3ddba7b046544571ad07ba"}
Nov 02 17:51:53.568938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.568849    4956 scope.go:122] "RemoveContainer" containerID="84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a"
Nov 02 17:51:53.569085 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.568938    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"
Nov 02 17:51:53.577519 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.577498    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tls-certs\") pod \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") "
Nov 02 17:51:53.577688 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.577678    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tmp-dir\") pod \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") "
Nov 02 17:51:53.577990 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.577964    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tmp-dir" pod "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" (UID: "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:53.578315 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.578300    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-model-cache\") pod \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") "
Nov 02 17:51:53.578557 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.578537    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-model-cache" pod "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" (UID: "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:53.578677 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.578661    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-kube-api-access-wvdqz\" (UniqueName: \"kubernetes.io/projected/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-kube-api-access-wvdqz\") pod \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") "
Nov 02 17:51:53.579063 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.579047    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-home\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-home\") pod \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") "
Nov 02 17:51:53.579300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.579284    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-dshm\") pod \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\" (UID: \"5a96f5ff-16d9-4d9d-a952-b1c5eea877ed\") "
Nov 02 17:51:53.580028 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.580007    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:53.580194 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.580180    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:53.580724 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.580684    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-home" pod "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" (UID: "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:53.581451 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.581410    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tls-certs" pod "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" (UID: "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:53.581972 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.581941    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-kube-api-access-wvdqz" pod "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" (UID: "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed"). InnerVolumeSpecName "kube-api-access-wvdqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:53.582315 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.582233    4956 scope.go:122] "RemoveContainer" containerID="84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a"
Nov 02 17:51:53.582614 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:51:53.582565    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a\": container with ID starting with 84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a not found: ID does not exist" containerID="84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a"
Nov 02 17:51:53.582614 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.582592    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a"} err="failed to get container status \"84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a\": rpc error: code = NotFound desc = could not find container \"84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a\": container with ID starting with 84e4c29c419e9a61572c502206945150f08ba2dd34274603b8e5b8b61640988a not found: ID does not exist"
Nov 02 17:51:53.582848 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.582836    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-dshm" pod "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" (UID: "5a96f5ff-16d9-4d9d-a952-b1c5eea877ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:53.681238 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.681153    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:53.681238 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.681191    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvdqz\" (UniqueName: \"kubernetes.io/projected/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-kube-api-access-wvdqz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:53.681238 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.681205    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:53.681238 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.681218    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:53.993204 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:53.993173    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"]
Nov 02 17:51:54.052482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:54.052441    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8cbedb9f/e2e-lifecycle-model-8cbedb9f-kserve-5d764cd9b5-qnccq"]
Nov 02 17:51:54.092025 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:54.091990    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 17:51:54.092214 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:54.092056    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 17:51:55.091895 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:55.091858    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 17:51:55.092434 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:55.091934    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 17:51:55.865542 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:55.865507    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" path="/var/lib/kubelet/pods/5a96f5ff-16d9-4d9d-a952-b1c5eea877ed/volumes"
Nov 02 17:51:56.125192 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:56.125067    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:56.126204 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:56.126177    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:56.126964 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:51:56.126934    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:51:59.057012 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:51:59.056956    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:52:18.995100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:18.995066    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"]
Nov 02 17:52:18.995878 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:18.995858    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" containerName="main"
Nov 02 17:52:18.995963 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:18.995880    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" containerName="main"
Nov 02 17:52:18.996034 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:18.996023    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a96f5ff-16d9-4d9d-a952-b1c5eea877ed" containerName="main"
Nov 02 17:52:18.999486 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:18.999470    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.076868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.076833    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077041 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.076884    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077041 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.077006    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077114 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.077056    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077238 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.077225    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.077280    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077327 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.077302    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077360 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.077333    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9clq\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-kube-api-access-v9clq\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.077479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.077466    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179017    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179259 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179144    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179259 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179178    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179259 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179208    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9clq\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-kube-api-access-v9clq\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179304    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179362    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179433 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179416    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179497    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179556    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.179831 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179713    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.180024 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.179998    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.180219 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.180085    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.180321 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.180135    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.180321 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.180229    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.181891 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.181852    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.182737 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.182713    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.203588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.203562    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"]
Nov 02 17:52:19.238158 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.238058    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.238333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.238237    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9clq\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-kube-api-access-v9clq\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.313365 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.313290    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:19.761426 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.761398    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"]
Nov 02 17:52:19.762407 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:52:19.762380    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d30825e_9ec5_4c5e_9f68_70ea2f00c849.slice/crio-a0f5c039c538b55cdf2e7ccc322bece844b3ab853bacaa667589491d2a68b262 WatchSource:0}: Error finding container a0f5c039c538b55cdf2e7ccc322bece844b3ab853bacaa667589491d2a68b262: Status 404 returned error can't find the container with id a0f5c039c538b55cdf2e7ccc322bece844b3ab853bacaa667589491d2a68b262
Nov 02 17:52:19.764989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.764954    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:19.765094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.765033    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:19.765094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:19.765078    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:20.763228 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:20.763194    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" event={"ID":"2d30825e-9ec5-4c5e-9f68-70ea2f00c849","Type":"ContainerStarted","Data":"0555aae36758e323a7406c2b6b1c9fbad21cd280f2578ff4cb8391d169af8ed7"}
Nov 02 17:52:20.763629 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:20.763249    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" event={"ID":"2d30825e-9ec5-4c5e-9f68-70ea2f00c849","Type":"ContainerStarted","Data":"a0f5c039c538b55cdf2e7ccc322bece844b3ab853bacaa667589491d2a68b262"}
Nov 02 17:52:21.314057 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:21.314018    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:21.315909 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:21.315869    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:52:21.316052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:21.315922    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 17:52:22.314503 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:22.314465    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:52:22.315096 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:22.315064    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 17:52:23.313950 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:23.313911    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:52:23.314300 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:23.314258    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 17:52:24.313685 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:24.313654    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:52:24.314156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:24.314114    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 17:52:25.008086 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.008028    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" podStartSLOduration=7.00801121 podStartE2EDuration="7.00801121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:20.99157438 +0000 UTC m=+1771.248604998" watchObservedRunningTime="2026-11-02 17:52:25.00801121 +0000 UTC m=+1775.265041833"
Nov 02 17:52:25.011179 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.011145    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"]
Nov 02 17:52:25.011377 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.011354    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy" containerID="cri-o://4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9" gracePeriod=30
Nov 02 17:52:25.014943 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.014919    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"]
Nov 02 17:52:25.015449 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.015420    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy" containerID="cri-o://e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08" gracePeriod=30
Nov 02 17:52:25.056815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.056772    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"]
Nov 02 17:52:25.057007 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.056973    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy" containerID="cri-o://a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e" gracePeriod=30
Nov 02 17:52:25.314338 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.314251    4956 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:52:25.314774 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:25.314748    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 17:52:26.714946 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:26.714901    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:26.715459 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:26.715373    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:26.716070 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:26.716048    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:52:29.053753 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:52:29.053714    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:52:30.368574 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.368542    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:52:30.375094 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.375065    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:52:30.422675 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.422651    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:52:30.468875 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.468838    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-envoy\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.469053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.468892    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72d3b545-2d27-417c-b815-d56360a399ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d3b545-2d27-417c-b815-d56360a399ed-istio-podinfo\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.469053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.468947    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-envoy\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.469053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.468986    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-data\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.469053 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469017    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-token\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-token\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.469304 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469067    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-credential-socket\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.469304 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469182    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istiod-ca-cert\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.469304 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469217    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d3b545-2d27-417c-b815-d56360a399ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d3b545-2d27-417c-b815-d56360a399ed-istiod-ca-cert\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.469464 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469365    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-data" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.469534 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469472    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-credential-socket" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.469666 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469641    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istiod-ca-cert" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:30.470011 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.469990    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-certs\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.470132 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470036    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-kube-api-access-v9ccl\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-kube-api-access-v9ccl\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.470132 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470081    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-istio-token\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-istio-token\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.470266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470138    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-podinfo\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.470266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470172    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-credential-socket\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.470266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470203    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-data\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.470266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470249    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afec1af-daec-419c-bdae-c6ec7e8d6452-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afec1af-daec-419c-bdae-c6ec7e8d6452-istiod-ca-cert\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.470489 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470298    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-data\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.470489 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470035    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d3b545-2d27-417c-b815-d56360a399ed-istiod-ca-cert" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:30.470489 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.470307    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-certs" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.471615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471381    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-data" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.471615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471524    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-data" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.471615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471562    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-credential-socket" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.471615 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471581    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afec1af-daec-419c-bdae-c6ec7e8d6452-istiod-ca-cert" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471630    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-socket\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471682    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-podinfo\") pod \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\" (UID: \"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac\") "
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471744    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-certs\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471791    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-kube-api-access-4s6nz\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-kube-api-access-4s6nz\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471827    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-credential-socket\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471935    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-token\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-token\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471970    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-socket\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.471992 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.471962    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-socket" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.472405 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.472214    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-credential-socket" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.473849 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.472482    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-socket" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.473849 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.472697    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-certs" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.473849 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.473045    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-envoy\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.473849 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.473079    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-certs\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.473849 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.473252    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-socket\") pod \"72d3b545-2d27-417c-b815-d56360a399ed\" (UID: \"72d3b545-2d27-417c-b815-d56360a399ed\") "
Nov 02 17:52:30.473849 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.473300    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-kube-api-access-rp9b7\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-kube-api-access-rp9b7\") pod \"9afec1af-daec-419c-bdae-c6ec7e8d6452\" (UID: \"9afec1af-daec-419c-bdae-c6ec7e8d6452\") "
Nov 02 17:52:30.474574 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.474549    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-token" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.474684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.474647    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-envoy" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.474759 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.474720    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72d3b545-2d27-417c-b815-d56360a399ed-istio-podinfo" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:30.474971 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.474859    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-certs" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.475044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.475023    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-envoy" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.475227    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-podinfo" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.475436    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-token" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.475544    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-istio-token" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476002    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476036    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476053    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476069    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476084    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476104    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476138    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72d3b545-2d27-417c-b815-d56360a399ed-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476154    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476169    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476188    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476203    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476218    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476233    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72d3b545-2d27-417c-b815-d56360a399ed-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476251    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476266    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476280    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476301    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476314    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afec1af-daec-419c-bdae-c6ec7e8d6452-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476326    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.477006 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.476923    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-kube-api-access-4s6nz" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "kube-api-access-4s6nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.478103 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.477131    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-kube-api-access-v9ccl" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "kube-api-access-v9ccl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.484734 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.484699    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-socket" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.485022 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.484913    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-podinfo" pod "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" (UID: "8b33a1ff-31e0-4a38-8036-9a46ed1fdcac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:30.486384 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.486358    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-kube-api-access-rp9b7" pod "9afec1af-daec-419c-bdae-c6ec7e8d6452" (UID: "9afec1af-daec-419c-bdae-c6ec7e8d6452"). InnerVolumeSpecName "kube-api-access-rp9b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.486752 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.486731    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-envoy" pod "72d3b545-2d27-417c-b815-d56360a399ed" (UID: "72d3b545-2d27-417c-b815-d56360a399ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.577939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577844    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9ccl\" (UniqueName: \"kubernetes.io/projected/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-kube-api-access-v9ccl\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.577939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577888    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.577939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577905    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.577939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577918    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s6nz\" (UniqueName: \"kubernetes.io/projected/72d3b545-2d27-417c-b815-d56360a399ed-kube-api-access-4s6nz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.577939 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577932    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.578331 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577946    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.578331 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577960    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72d3b545-2d27-417c-b815-d56360a399ed-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.578331 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.577973    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9b7\" (UniqueName: \"kubernetes.io/projected/9afec1af-daec-419c-bdae-c6ec7e8d6452-kube-api-access-rp9b7\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.853608 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.853512    4956 generic.go:356] "Generic (PLEG): container finished" podID="72d3b545-2d27-417c-b815-d56360a399ed" containerID="e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08" exitCode=0
Nov 02 17:52:30.853608 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.853579    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"
Nov 02 17:52:30.853608 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.853589    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" event={"ID":"72d3b545-2d27-417c-b815-d56360a399ed","Type":"ContainerDied","Data":"e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08"}
Nov 02 17:52:30.853907 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.853677    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5" event={"ID":"72d3b545-2d27-417c-b815-d56360a399ed","Type":"ContainerDied","Data":"5c1a49c9086dc11f518c703dceae745082767e6104d2e69f2f22a476d14a11b2"}
Nov 02 17:52:30.853907 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.853708    4956 scope.go:122] "RemoveContainer" containerID="e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08"
Nov 02 17:52:30.855413 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.855392    4956 generic.go:356] "Generic (PLEG): container finished" podID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerID="4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9" exitCode=0
Nov 02 17:52:30.858160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.855525    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" event={"ID":"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac","Type":"ContainerDied","Data":"4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9"}
Nov 02 17:52:30.858160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.855573    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn" event={"ID":"8b33a1ff-31e0-4a38-8036-9a46ed1fdcac","Type":"ContainerDied","Data":"632dbe4e16eb0017c5add94ae41d30e9585a33646c43235697e3922fa7bb7bc7"}
Nov 02 17:52:30.858160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.855648    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"
Nov 02 17:52:30.862557 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.862532    4956 generic.go:356] "Generic (PLEG): container finished" podID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerID="a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e" exitCode=0
Nov 02 17:52:30.862663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.862568    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" event={"ID":"9afec1af-daec-419c-bdae-c6ec7e8d6452","Type":"ContainerDied","Data":"a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e"}
Nov 02 17:52:30.862663 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.862598    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"
Nov 02 17:52:30.862763 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.862602    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr" event={"ID":"9afec1af-daec-419c-bdae-c6ec7e8d6452","Type":"ContainerDied","Data":"2d6c21e2cd81cdb579409d1712a49f8613b4de2dd73f96a87611e41b026ef732"}
Nov 02 17:52:30.868244 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.868228    4956 scope.go:122] "RemoveContainer" containerID="e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08"
Nov 02 17:52:30.868697 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:52:30.868546    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08\": container with ID starting with e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08 not found: ID does not exist" containerID="e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08"
Nov 02 17:52:30.868697 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.868585    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08"} err="failed to get container status \"e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08\": rpc error: code = NotFound desc = could not find container \"e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08\": container with ID starting with e232e92ce945ccf19cfbfbc253fd23b1d41554b12e7ecf47a053b2e41af78c08 not found: ID does not exist"
Nov 02 17:52:30.868697 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.868609    4956 scope.go:122] "RemoveContainer" containerID="4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9"
Nov 02 17:52:30.879438 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.879390    4956 scope.go:122] "RemoveContainer" containerID="4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9"
Nov 02 17:52:30.879645 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:52:30.879628    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9\": container with ID starting with 4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9 not found: ID does not exist" containerID="4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9"
Nov 02 17:52:30.879702 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.879654    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9"} err="failed to get container status \"4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9\": rpc error: code = NotFound desc = could not find container \"4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9\": container with ID starting with 4968ddb5d7f5cb8aba6b3e3658edc0bf90293005b07b8dc102e4a3431a00e0a9 not found: ID does not exist"
Nov 02 17:52:30.879702 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.879674    4956 scope.go:122] "RemoveContainer" containerID="a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e"
Nov 02 17:52:30.888918 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.888903    4956 scope.go:122] "RemoveContainer" containerID="a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e"
Nov 02 17:52:30.889224 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:52:30.889203    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e\": container with ID starting with a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e not found: ID does not exist" containerID="a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e"
Nov 02 17:52:30.889290 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:30.889233    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e"} err="failed to get container status \"a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e\": rpc error: code = NotFound desc = could not find container \"a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e\": container with ID starting with a43ac9712b486d07773364fa21145a9062855ed8077655144d01ba43d5b12a3e not found: ID does not exist"
Nov 02 17:52:31.020877 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.020841    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"]
Nov 02 17:52:31.025066 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.025038    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-76vx5"]
Nov 02 17:52:31.134180 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.134065    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"]
Nov 02 17:52:31.137476 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.137444    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-pn6fr"]
Nov 02 17:52:31.245757 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.245727    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"]
Nov 02 17:52:31.254648 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.254618    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-k6lnn"]
Nov 02 17:52:31.865691 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.865655    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d3b545-2d27-417c-b815-d56360a399ed" path="/var/lib/kubelet/pods/72d3b545-2d27-417c-b815-d56360a399ed/volumes"
Nov 02 17:52:31.866283 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.866258    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" path="/var/lib/kubelet/pods/8b33a1ff-31e0-4a38-8036-9a46ed1fdcac/volumes"
Nov 02 17:52:31.866769 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:31.866754    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" path="/var/lib/kubelet/pods/9afec1af-daec-419c-bdae-c6ec7e8d6452/volumes"
Nov 02 17:52:39.285478 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.285436    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"]
Nov 02 17:52:39.286041 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286027    4956 cpu_manager.go:414] "containerMap: removing container" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy"
Nov 02 17:52:39.286087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286042    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy"
Nov 02 17:52:39.286087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286063    4956 cpu_manager.go:414] "containerMap: removing container" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy"
Nov 02 17:52:39.286087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286069    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy"
Nov 02 17:52:39.286087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286082    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy"
Nov 02 17:52:39.286087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286087    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy"
Nov 02 17:52:39.286255 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286191    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b33a1ff-31e0-4a38-8036-9a46ed1fdcac" containerName="istio-proxy"
Nov 02 17:52:39.286255 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286201    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d3b545-2d27-417c-b815-d56360a399ed" containerName="istio-proxy"
Nov 02 17:52:39.286255 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.286208    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afec1af-daec-419c-bdae-c6ec7e8d6452" containerName="istio-proxy"
Nov 02 17:52:39.291543 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.291519    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:39.345884 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.345654    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5bf666\""
Nov 02 17:52:39.395518 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.395488    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5bf666-dockercfg-7n5fg\""
Nov 02 17:52:39.399671 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.399633    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7f52c-7708-4c5f-97b8-029e426f9c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:39.400298 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.400267    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rqx\" (UniqueName: \"kubernetes.io/projected/63c7f52c-7708-4c5f-97b8-029e426f9c0f-kube-api-access-s8rqx\") pod \"payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:39.419177 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.419135    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"]
Nov 02 17:52:39.502506 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.502473    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7f52c-7708-4c5f-97b8-029e426f9c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:39.502677 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.502557    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rqx\" (UniqueName: \"kubernetes.io/projected/63c7f52c-7708-4c5f-97b8-029e426f9c0f-kube-api-access-s8rqx\") pod \"payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:39.503266 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.503247    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7f52c-7708-4c5f-97b8-029e426f9c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:39.570975 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.570886    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rqx\" (UniqueName: \"kubernetes.io/projected/63c7f52c-7708-4c5f-97b8-029e426f9c0f-kube-api-access-s8rqx\") pod \"payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:39.606550 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:39.605827    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:40.006807 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:40.006769    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"]
Nov 02 17:52:40.930991 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:40.930944    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z" event={"ID":"63c7f52c-7708-4c5f-97b8-029e426f9c0f","Type":"ContainerStarted","Data":"036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316"}
Nov 02 17:52:40.931446 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:40.931009    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z" event={"ID":"63c7f52c-7708-4c5f-97b8-029e426f9c0f","Type":"ContainerStarted","Data":"f735d5642b9d727f46c6df4f59d08223b435209ef7e396fd6acb6d735c09385b"}
Nov 02 17:52:40.931446 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:40.931049    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:41.049105 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:41.049037    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z" podStartSLOduration=2.049016539 podStartE2EDuration="2.049016539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:41.044213117 +0000 UTC m=+1791.301243737" watchObservedRunningTime="2026-11-02 17:52:41.049016539 +0000 UTC m=+1791.306047161"
Nov 02 17:52:50.092824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:50.092796    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:52:50.092824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:50.092827    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:52:51.949570 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:52:51.949536    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:52:59.054609 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:52:59.054572    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:53:29.055477 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:53:29.055438    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:53:50.639833 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:50.639800    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"]
Nov 02 17:53:50.640427 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:50.640042    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" containerID="cri-o://975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb" gracePeriod=30
Nov 02 17:53:52.860470 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:52.860435    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:53:52.860885 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:52.860522    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:53:55.907780 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.907707    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:53:55.954010 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.953981    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-certs\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954205 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954038    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-socket\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954205 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954170    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-data\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954205 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954199    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-credential-socket\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954331 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954247    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-kube-api-access-jsh7g\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-kube-api-access-jsh7g\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954331 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954303    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-certs" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:55.954422 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954328    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-envoy\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954422 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954375    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-token\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-token\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954517 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954397    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-socket" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:55.954517 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954479    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-data" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:55.954517 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954494    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-podinfo\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954517 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954500    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-credential-socket" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:55.954745 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954534    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbdfc53-0624-421c-82e0-6b083865f6fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbdfc53-0624-421c-82e0-6b083865f6fd-istiod-ca-cert\") pod \"2bbdfc53-0624-421c-82e0-6b083865f6fd\" (UID: \"2bbdfc53-0624-421c-82e0-6b083865f6fd\") "
Nov 02 17:53:55.954955 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.954931    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbdfc53-0624-421c-82e0-6b083865f6fd-istiod-ca-cert" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:55.955588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.955460    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbdfc53-0624-421c-82e0-6b083865f6fd-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:55.955588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.955488    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:55.955588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.955503    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:55.955588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.955517    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:55.955588 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.955533    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:55.956751 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.956724    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-envoy" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:55.956845 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.956804    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-token" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:55.956845 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.956819    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-kube-api-access-jsh7g" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "kube-api-access-jsh7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:55.956932 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:55.956910    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-podinfo" pod "2bbdfc53-0624-421c-82e0-6b083865f6fd" (UID: "2bbdfc53-0624-421c-82e0-6b083865f6fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:53:56.056643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.056609    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:56.056643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.056635    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:56.056643 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.056646    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsh7g\" (UniqueName: \"kubernetes.io/projected/2bbdfc53-0624-421c-82e0-6b083865f6fd-kube-api-access-jsh7g\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:56.056871 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.056655    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbdfc53-0624-421c-82e0-6b083865f6fd-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:56.465149 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.465076    4956 generic.go:356] "Generic (PLEG): container finished" podID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerID="975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb" exitCode=0
Nov 02 17:53:56.465355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.465171    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" event={"ID":"2bbdfc53-0624-421c-82e0-6b083865f6fd","Type":"ContainerDied","Data":"975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb"}
Nov 02 17:53:56.465355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.465223    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s" event={"ID":"2bbdfc53-0624-421c-82e0-6b083865f6fd","Type":"ContainerDied","Data":"6e791d626bba140c0f51d0eb429e70a7f584b6c1101987c889f978936cfe67d2"}
Nov 02 17:53:56.465355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.465253    4956 scope.go:122] "RemoveContainer" containerID="975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb"
Nov 02 17:53:56.465355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.465173    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"
Nov 02 17:53:56.476455 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.476432    4956 scope.go:122] "RemoveContainer" containerID="975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb"
Nov 02 17:53:56.476825 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:53:56.476803    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb\": container with ID starting with 975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb not found: ID does not exist" containerID="975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb"
Nov 02 17:53:56.476900 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.476836    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb"} err="failed to get container status \"975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb\": rpc error: code = NotFound desc = could not find container \"975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb\": container with ID starting with 975f299e8f21b2999c576c286d7ff3e909bab953159b29efe02f3b58fad0dbfb not found: ID does not exist"
Nov 02 17:53:56.630112 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.630058    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"]
Nov 02 17:53:56.679922 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:56.679883    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-xw72s"]
Nov 02 17:53:57.873898 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:53:57.873860    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" path="/var/lib/kubelet/pods/2bbdfc53-0624-421c-82e0-6b083865f6fd/volumes"
Nov 02 17:53:59.058987 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:53:59.058948    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:54:05.760909 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:05.760866    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"]
Nov 02 17:54:05.761508 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:05.761163    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" containerID="cri-o://a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981" gracePeriod=30
Nov 02 17:54:08.682459 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:08.682410    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:54:08.682953 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:08.682523    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:54:11.059845 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.059814    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:54:11.151709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151672    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-credential-socket\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.151709 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151714    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-token\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-token\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.151954 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151758    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-certs\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.151954 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151837    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-podinfo\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.151954 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151856    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istiod-ca-cert\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.151954 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151927    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-envoy\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.152218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151992    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-data\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.152218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.151992    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-credential-socket" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:11.152218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152077    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-socket\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.152218 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152173    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-kube-api-access-sh8qj\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-kube-api-access-sh8qj\") pod \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\" (UID: \"491d7fbc-9e54-48c8-b5aa-17bb0a948eed\") "
Nov 02 17:54:11.152429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152233    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-certs" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:11.152429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152336    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istiod-ca-cert" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:11.152429 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152346    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-data" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:11.152580 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152462    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-socket" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:11.152801 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152772    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.152801 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152797    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.152938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152813    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.152938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152825    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.152938 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.152839    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.154355 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.154332    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-envoy" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:11.154457 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.154434    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-podinfo" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:11.154752 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.154721    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-token" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:11.154851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.154800    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-kube-api-access-sh8qj" pod "491d7fbc-9e54-48c8-b5aa-17bb0a948eed" (UID: "491d7fbc-9e54-48c8-b5aa-17bb0a948eed"). InnerVolumeSpecName "kube-api-access-sh8qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:11.253897 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.253866    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.253897 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.253894    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh8qj\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-kube-api-access-sh8qj\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.254091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.253906    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.254091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.253915    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491d7fbc-9e54-48c8-b5aa-17bb0a948eed-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:11.574610 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.574577    4956 generic.go:356] "Generic (PLEG): container finished" podID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerID="a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981" exitCode=0
Nov 02 17:54:11.574795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.574616    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" event={"ID":"491d7fbc-9e54-48c8-b5aa-17bb0a948eed","Type":"ContainerDied","Data":"a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981"}
Nov 02 17:54:11.574795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.574666    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"
Nov 02 17:54:11.574795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.574689    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt" event={"ID":"491d7fbc-9e54-48c8-b5aa-17bb0a948eed","Type":"ContainerDied","Data":"284fa93a452aa393d6f69df1c479914a3b1f9eb1c4ab17701863f8b54a43a2f7"}
Nov 02 17:54:11.574795 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.574715    4956 scope.go:122] "RemoveContainer" containerID="a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981"
Nov 02 17:54:11.586078 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.586055    4956 scope.go:122] "RemoveContainer" containerID="a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981"
Nov 02 17:54:11.586390 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:54:11.586364    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981\": container with ID starting with a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981 not found: ID does not exist" containerID="a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981"
Nov 02 17:54:11.586462 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.586404    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981"} err="failed to get container status \"a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981\": rpc error: code = NotFound desc = could not find container \"a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981\": container with ID starting with a272349505e2edf41fadfde5a187d2993b482060b35024b3a30006ab207c9981 not found: ID does not exist"
Nov 02 17:54:11.745398 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.745363    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"]
Nov 02 17:54:11.793644 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.793602    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-hfflt"]
Nov 02 17:54:11.865487 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:54:11.865406    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" path="/var/lib/kubelet/pods/491d7fbc-9e54-48c8-b5aa-17bb0a948eed/volumes"
Nov 02 17:54:29.053257 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:54:29.053221    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:54:59.054797 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:54:59.054756    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:55:29.058578 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:55:29.058539    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:55:32.949815 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:32.949784    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"]
Nov 02 17:55:32.952195 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:32.949985    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy" containerID="cri-o://0555aae36758e323a7406c2b6b1c9fbad21cd280f2578ff4cb8391d169af8ed7" gracePeriod=30
Nov 02 17:55:32.962657 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:32.962629    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"]
Nov 02 17:55:32.962840 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:32.962819    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy" containerID="cri-o://2500cb3a4027b0dba50689d522dcac578b63209bc29de3661fc28cd389b850e1" gracePeriod=30
Nov 02 17:55:38.173712 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.173679    4956 generic.go:356] "Generic (PLEG): container finished" podID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerID="2500cb3a4027b0dba50689d522dcac578b63209bc29de3661fc28cd389b850e1" exitCode=0
Nov 02 17:55:38.174233 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.173729    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" event={"ID":"f4d080ee-bf15-4119-83f0-b3cfc14a3c73","Type":"ContainerDied","Data":"2500cb3a4027b0dba50689d522dcac578b63209bc29de3661fc28cd389b850e1"}
Nov 02 17:55:38.175920 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.175894    4956 generic.go:356] "Generic (PLEG): container finished" podID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerID="0555aae36758e323a7406c2b6b1c9fbad21cd280f2578ff4cb8391d169af8ed7" exitCode=0
Nov 02 17:55:38.176060 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.176039    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" event={"ID":"2d30825e-9ec5-4c5e-9f68-70ea2f00c849","Type":"ContainerDied","Data":"0555aae36758e323a7406c2b6b1c9fbad21cd280f2578ff4cb8391d169af8ed7"}
Nov 02 17:55:38.271853 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.271818    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:55:38.298303 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.298282    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:55:38.352434 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352401    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-socket\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.352636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352452    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-socket\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.352636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352474    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-data\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.352636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352525    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-token\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-token\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.352636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352547    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istiod-ca-cert\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.352636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352588    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-credential-socket\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.352636 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352634    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-podinfo\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352667    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-credential-socket\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352729    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-socket" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352735    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-socket" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352742    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-token\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-token\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352811    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-data" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352829    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-envoy\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352865    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-data\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352892    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-kube-api-access-qhf27\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-kube-api-access-qhf27\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352922    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-certs\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.352950    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istiod-ca-cert\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353005    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-certs\") pod \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\" (UID: \"f4d080ee-bf15-4119-83f0-b3cfc14a3c73\") "
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353014    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-credential-socket" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.353023 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353021    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istiod-ca-cert" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:38.353969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353172    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-kube-api-access-v9clq\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-kube-api-access-v9clq\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.353969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353232    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-podinfo\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.353969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353276    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-envoy\") pod \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\" (UID: \"2d30825e-9ec5-4c5e-9f68-70ea2f00c849\") "
Nov 02 17:55:38.353969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353309    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-certs" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.353969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353590    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istiod-ca-cert" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:38.353969 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.353909    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-data" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.354308 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354107    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-credential-socket" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.354308 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354210    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-certs" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.354430 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354361    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354430 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354384    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354430 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354409    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354430 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354424    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354439    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354453    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354468    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354482    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354497    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.354626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.354511    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.355641 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.355601    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-kube-api-access-qhf27" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "kube-api-access-qhf27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.355875 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.355699    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-envoy" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.355875 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.355766    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-token" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.356142 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.356054    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-token" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.356225 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.356205    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-kube-api-access-v9clq" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "kube-api-access-v9clq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.356293 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.356241    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-envoy" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.356641 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.356624    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-podinfo" pod "2d30825e-9ec5-4c5e-9f68-70ea2f00c849" (UID: "2d30825e-9ec5-4c5e-9f68-70ea2f00c849"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:38.357332 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.357308    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-podinfo" pod "f4d080ee-bf15-4119-83f0-b3cfc14a3c73" (UID: "f4d080ee-bf15-4119-83f0-b3cfc14a3c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:38.455603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455566    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.455603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455597    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.455603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455607    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.455603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455615    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.455968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455624    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.455968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455632    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.455968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455641    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhf27\" (UniqueName: \"kubernetes.io/projected/f4d080ee-bf15-4119-83f0-b3cfc14a3c73-kube-api-access-qhf27\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.455968 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:38.455651    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9clq\" (UniqueName: \"kubernetes.io/projected/2d30825e-9ec5-4c5e-9f68-70ea2f00c849-kube-api-access-v9clq\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:39.183344 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.183309    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"
Nov 02 17:55:39.183806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.183348    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q" event={"ID":"2d30825e-9ec5-4c5e-9f68-70ea2f00c849","Type":"ContainerDied","Data":"a0f5c039c538b55cdf2e7ccc322bece844b3ab853bacaa667589491d2a68b262"}
Nov 02 17:55:39.183806 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.183418    4956 scope.go:122] "RemoveContainer" containerID="0555aae36758e323a7406c2b6b1c9fbad21cd280f2578ff4cb8391d169af8ed7"
Nov 02 17:55:39.185080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.185058    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"
Nov 02 17:55:39.185185 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.185062    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc" event={"ID":"f4d080ee-bf15-4119-83f0-b3cfc14a3c73","Type":"ContainerDied","Data":"27d6f8960a5052ed52d36ffe5a3c420ea3b50e559ea396767e12c5025c7beae7"}
Nov 02 17:55:39.195513 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.195495    4956 scope.go:122] "RemoveContainer" containerID="2500cb3a4027b0dba50689d522dcac578b63209bc29de3661fc28cd389b850e1"
Nov 02 17:55:39.347689 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.347597    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"]
Nov 02 17:55:39.397457 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.397421    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-fm5gc"]
Nov 02 17:55:39.507996 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.507953    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"]
Nov 02 17:55:39.510722 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.510701    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-k9p4q"]
Nov 02 17:55:39.866061 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.866027    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" path="/var/lib/kubelet/pods/2d30825e-9ec5-4c5e-9f68-70ea2f00c849/volumes"
Nov 02 17:55:39.866590 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:39.866574    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" path="/var/lib/kubelet/pods/f4d080ee-bf15-4119-83f0-b3cfc14a3c73/volumes"
Nov 02 17:55:41.702075 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:41.702037    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"]
Nov 02 17:55:41.702551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:41.702259    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864" podUID="c447e80a-d759-4cc7-8913-9e101b649d1e" containerName="maas-api" containerID="cri-o://85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485" gracePeriod=30
Nov 02 17:55:42.007319 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.007293    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:55:42.101803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.101772    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c447e80a-d759-4cc7-8913-9e101b649d1e-kube-api-access-lb6qg\" (UniqueName: \"kubernetes.io/projected/c447e80a-d759-4cc7-8913-9e101b649d1e-kube-api-access-lb6qg\") pod \"c447e80a-d759-4cc7-8913-9e101b649d1e\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") "
Nov 02 17:55:42.101972 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.101842    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls\") pod \"c447e80a-d759-4cc7-8913-9e101b649d1e\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") "
Nov 02 17:55:42.101972 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.101919    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-metrics-certs\") pod \"c447e80a-d759-4cc7-8913-9e101b649d1e\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") "
Nov 02 17:55:42.101972 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.101945    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c447e80a-d759-4cc7-8913-9e101b649d1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c447e80a-d759-4cc7-8913-9e101b649d1e-openshift-service-ca\") pod \"c447e80a-d759-4cc7-8913-9e101b649d1e\" (UID: \"c447e80a-d759-4cc7-8913-9e101b649d1e\") "
Nov 02 17:55:42.102453 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.102423    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c447e80a-d759-4cc7-8913-9e101b649d1e-openshift-service-ca" pod "c447e80a-d759-4cc7-8913-9e101b649d1e" (UID: "c447e80a-d759-4cc7-8913-9e101b649d1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:42.104223 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.104196    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls" pod "c447e80a-d759-4cc7-8913-9e101b649d1e" (UID: "c447e80a-d759-4cc7-8913-9e101b649d1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:42.104327 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.104267    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c447e80a-d759-4cc7-8913-9e101b649d1e-kube-api-access-lb6qg" pod "c447e80a-d759-4cc7-8913-9e101b649d1e" (UID: "c447e80a-d759-4cc7-8913-9e101b649d1e"). InnerVolumeSpecName "kube-api-access-lb6qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:42.104327 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.104267    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-metrics-certs" pod "c447e80a-d759-4cc7-8913-9e101b649d1e" (UID: "c447e80a-d759-4cc7-8913-9e101b649d1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:42.203658 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.203624    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb6qg\" (UniqueName: \"kubernetes.io/projected/c447e80a-d759-4cc7-8913-9e101b649d1e-kube-api-access-lb6qg\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:42.203658 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.203651    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:42.203658 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.203664    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c447e80a-d759-4cc7-8913-9e101b649d1e-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:42.203981 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.203674    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c447e80a-d759-4cc7-8913-9e101b649d1e-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:42.210093 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.210063    4956 generic.go:356] "Generic (PLEG): container finished" podID="c447e80a-d759-4cc7-8913-9e101b649d1e" containerID="85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485" exitCode=0
Nov 02 17:55:42.210292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.210097    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864" event={"ID":"c447e80a-d759-4cc7-8913-9e101b649d1e","Type":"ContainerDied","Data":"85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485"}
Nov 02 17:55:42.210292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.210159    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"
Nov 02 17:55:42.210292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.210198    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864" event={"ID":"c447e80a-d759-4cc7-8913-9e101b649d1e","Type":"ContainerDied","Data":"9fd80801d44200f96fdec5d00f6e5f3c3e267317025b3ed71f7bd80bc7a128e5"}
Nov 02 17:55:42.210292 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.210224    4956 scope.go:122] "RemoveContainer" containerID="85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485"
Nov 02 17:55:42.225350 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.225322    4956 scope.go:122] "RemoveContainer" containerID="85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485"
Nov 02 17:55:42.225738 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:55:42.225715    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485\": container with ID starting with 85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485 not found: ID does not exist" containerID="85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485"
Nov 02 17:55:42.225828 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.225751    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485"} err="failed to get container status \"85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485\": rpc error: code = NotFound desc = could not find container \"85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485\": container with ID starting with 85e8a1389327515664a118d80e42a6e341191ed29e729e899d317a4e95613485 not found: ID does not exist"
Nov 02 17:55:42.374824 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.374789    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"]
Nov 02 17:55:42.378160 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:42.378105    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fa615-99f79cdb5-fh864"]
Nov 02 17:55:43.865657 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:43.865619    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c447e80a-d759-4cc7-8913-9e101b649d1e" path="/var/lib/kubelet/pods/c447e80a-d759-4cc7-8913-9e101b649d1e/volumes"
Nov 02 17:55:50.288567 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:55:50.288536    4956 scope.go:122] "RemoveContainer" containerID="1ece4095335f9b0779ca04b78e88224c86b88bc1e93bca094ab554e5001ad206"
Nov 02 17:55:59.053959 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:55:59.053920    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:56:29.053683 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:56:29.053636    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:56:32.573428 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.573239    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"]
Nov 02 17:56:32.574558 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574532    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c447e80a-d759-4cc7-8913-9e101b649d1e" containerName="maas-api"
Nov 02 17:56:32.574731 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574718    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c447e80a-d759-4cc7-8913-9e101b649d1e" containerName="maas-api"
Nov 02 17:56:32.574845 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574834    4956 cpu_manager.go:414] "containerMap: removing container" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy"
Nov 02 17:56:32.574935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574916    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy"
Nov 02 17:56:32.574935 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574940    4956 cpu_manager.go:414] "containerMap: removing container" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy"
Nov 02 17:56:32.575156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574948    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy"
Nov 02 17:56:32.575156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574982    4956 cpu_manager.go:414] "containerMap: removing container" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy"
Nov 02 17:56:32.575156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.574989    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy"
Nov 02 17:56:32.575156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.575005    4956 cpu_manager.go:414] "containerMap: removing container" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy"
Nov 02 17:56:32.575156 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.575013    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy"
Nov 02 17:56:32.575400 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.575198    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d080ee-bf15-4119-83f0-b3cfc14a3c73" containerName="istio-proxy"
Nov 02 17:56:32.575400 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.575214    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbdfc53-0624-421c-82e0-6b083865f6fd" containerName="istio-proxy"
Nov 02 17:56:32.575400 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.575225    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d30825e-9ec5-4c5e-9f68-70ea2f00c849" containerName="istio-proxy"
Nov 02 17:56:32.575400 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.575244    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="491d7fbc-9e54-48c8-b5aa-17bb0a948eed" containerName="istio-proxy"
Nov 02 17:56:32.575400 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.575255    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c447e80a-d759-4cc7-8913-9e101b649d1e" containerName="maas-api"
Nov 02 17:56:32.579655 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.579636    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:32.629787 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.629750    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"]
Nov 02 17:56:32.671915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.671886    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e6cb24-e10d-423d-8740-a7ebbc0a1138-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:32.672085 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.672052    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhnm\" (UniqueName: \"kubernetes.io/projected/24e6cb24-e10d-423d-8740-a7ebbc0a1138-kube-api-access-5fhnm\") pod \"payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:32.675334 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.675299    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5a1e27\""
Nov 02 17:56:32.675466 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.675407    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5a1e27-dockercfg-prbxf\""
Nov 02 17:56:32.773864 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.773832    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e6cb24-e10d-423d-8740-a7ebbc0a1138-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:32.774050 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.773942    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhnm\" (UniqueName: \"kubernetes.io/projected/24e6cb24-e10d-423d-8740-a7ebbc0a1138-kube-api-access-5fhnm\") pod \"payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:32.774626 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.774605    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e6cb24-e10d-423d-8740-a7ebbc0a1138-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:32.831911 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.831836    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhnm\" (UniqueName: \"kubernetes.io/projected/24e6cb24-e10d-423d-8740-a7ebbc0a1138-kube-api-access-5fhnm\") pod \"payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:32.903666 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:32.903625    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:33.276548 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:33.276519    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"]
Nov 02 17:56:33.278445 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:56:33.278416    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e6cb24_e10d_423d_8740_a7ebbc0a1138.slice/crio-42663da80f27550d971166cf845e82bb771cccb5c585cda77cac4313ecee7cc7 WatchSource:0}: Error finding container 42663da80f27550d971166cf845e82bb771cccb5c585cda77cac4313ecee7cc7: Status 404 returned error can't find the container with id 42663da80f27550d971166cf845e82bb771cccb5c585cda77cac4313ecee7cc7
Nov 02 17:56:33.578919 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:33.578826    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b" event={"ID":"24e6cb24-e10d-423d-8740-a7ebbc0a1138","Type":"ContainerStarted","Data":"b8ec7d619083000f24ee2f8154a17913bec32018f2681f74b531a94a8f22df1f"}
Nov 02 17:56:33.578919 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:33.578876    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b" event={"ID":"24e6cb24-e10d-423d-8740-a7ebbc0a1138","Type":"ContainerStarted","Data":"42663da80f27550d971166cf845e82bb771cccb5c585cda77cac4313ecee7cc7"}
Nov 02 17:56:33.579381 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:33.578951    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:33.695233 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:33.695178    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b" podStartSLOduration=1.6951613060000001 podStartE2EDuration="1.695161306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:33.692202316 +0000 UTC m=+2023.949232930" watchObservedRunningTime="2026-11-02 17:56:33.695161306 +0000 UTC m=+2023.952191927"
Nov 02 17:56:44.588518 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:56:44.588484    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 17:56:59.053385 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:56:59.053341    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:57:21.599230 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:21.599193    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"]
Nov 02 17:57:26.599753 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.599684    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z" podUID="63c7f52c-7708-4c5f-97b8-029e426f9c0f" containerName="payload-pre-processing" containerID="cri-o://036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316" gracePeriod=25
Nov 02 17:57:26.877603 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.877563    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:57:26.913887 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.913849    4956 generic.go:356] "Generic (PLEG): container finished" podID="63c7f52c-7708-4c5f-97b8-029e426f9c0f" containerID="036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316" exitCode=0
Nov 02 17:57:26.914091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.913913    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"
Nov 02 17:57:26.914091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.913941    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z" event={"ID":"63c7f52c-7708-4c5f-97b8-029e426f9c0f","Type":"ContainerDied","Data":"036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316"}
Nov 02 17:57:26.914091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.914009    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z" event={"ID":"63c7f52c-7708-4c5f-97b8-029e426f9c0f","Type":"ContainerDied","Data":"f735d5642b9d727f46c6df4f59d08223b435209ef7e396fd6acb6d735c09385b"}
Nov 02 17:57:26.914091 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.914040    4956 scope.go:122] "RemoveContainer" containerID="036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316"
Nov 02 17:57:26.924901 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.924878    4956 scope.go:122] "RemoveContainer" containerID="036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316"
Nov 02 17:57:26.925257 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:57:26.925239    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316\": container with ID starting with 036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316 not found: ID does not exist" containerID="036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316"
Nov 02 17:57:26.925333 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.925266    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316"} err="failed to get container status \"036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316\": rpc error: code = NotFound desc = could not find container \"036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316\": container with ID starting with 036fa83447ff988639e3ed2a68e74fdeefdf78dde8ca64ceba6e4ed217a8f316 not found: ID does not exist"
Nov 02 17:57:26.980482 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.980452    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c7f52c-7708-4c5f-97b8-029e426f9c0f-kube-api-access-s8rqx\" (UniqueName: \"kubernetes.io/projected/63c7f52c-7708-4c5f-97b8-029e426f9c0f-kube-api-access-s8rqx\") pod \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") "
Nov 02 17:57:26.980660 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.980538    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c7f52c-7708-4c5f-97b8-029e426f9c0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7f52c-7708-4c5f-97b8-029e426f9c0f-plugins-config-volume\") pod \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\" (UID: \"63c7f52c-7708-4c5f-97b8-029e426f9c0f\") "
Nov 02 17:57:26.980949 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.980928    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c7f52c-7708-4c5f-97b8-029e426f9c0f-plugins-config-volume" pod "63c7f52c-7708-4c5f-97b8-029e426f9c0f" (UID: "63c7f52c-7708-4c5f-97b8-029e426f9c0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:26.981087 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.981071    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7f52c-7708-4c5f-97b8-029e426f9c0f-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:26.982690 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:26.982664    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c7f52c-7708-4c5f-97b8-029e426f9c0f-kube-api-access-s8rqx" pod "63c7f52c-7708-4c5f-97b8-029e426f9c0f" (UID: "63c7f52c-7708-4c5f-97b8-029e426f9c0f"). InnerVolumeSpecName "kube-api-access-s8rqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:27.082750 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:27.082706    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8rqx\" (UniqueName: \"kubernetes.io/projected/63c7f52c-7708-4c5f-97b8-029e426f9c0f-kube-api-access-s8rqx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:27.336136 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:27.336083    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"]
Nov 02 17:57:27.338931 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:27.338905    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5bf666-7f98ccfc95-ckt6z"]
Nov 02 17:57:27.864995 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:27.864961    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c7f52c-7708-4c5f-97b8-029e426f9c0f" path="/var/lib/kubelet/pods/63c7f52c-7708-4c5f-97b8-029e426f9c0f/volumes"
Nov 02 17:57:29.053750 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:57:29.053712    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:57:44.247763 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:44.247717    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"]
Nov 02 17:57:44.248280 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:44.247985    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" containerID="cri-o://87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0" gracePeriod=30
Nov 02 17:57:47.934213 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:47.934183    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:57:47.934637 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:47.934264    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:57:49.509345 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.509318    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:57:49.559850 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.559753    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-data\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.559850 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.559798    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-token\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-token\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.559855    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-podinfo\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.559899    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-socket\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.559996    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-kube-api-access-vrlwr\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-kube-api-access-vrlwr\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560023    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-credential-socket\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560080 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560070    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-envoy\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560360 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560067    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-data" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:49.560360 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560174    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1abc52b-f034-464d-87d5-ceb44b55255d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1abc52b-f034-464d-87d5-ceb44b55255d-istiod-ca-cert\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560360 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560233    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-certs\") pod \"e1abc52b-f034-464d-87d5-ceb44b55255d\" (UID: \"e1abc52b-f034-464d-87d5-ceb44b55255d\") "
Nov 02 17:57:49.560360 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560319    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-socket" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:49.560586 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560410    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-credential-socket" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:49.560586 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560580    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-certs" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:49.560690 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.560634    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1abc52b-f034-464d-87d5-ceb44b55255d-istiod-ca-cert" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:49.561098 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.561067    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.561098 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.561091    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.561098 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.561106    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.561304 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.561135    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1abc52b-f034-464d-87d5-ceb44b55255d-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.561304 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.561150    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.562954 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.562931    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-token" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:49.562954 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.562944    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-podinfo" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:49.563103 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.562969    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-envoy" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:49.563178 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.563112    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-kube-api-access-vrlwr" pod "e1abc52b-f034-464d-87d5-ceb44b55255d" (UID: "e1abc52b-f034-464d-87d5-ceb44b55255d"). InnerVolumeSpecName "kube-api-access-vrlwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:49.662873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.662837    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrlwr\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-kube-api-access-vrlwr\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.662873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.662868    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.662873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.662880    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.663112 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.662888    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1abc52b-f034-464d-87d5-ceb44b55255d-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:49.880714 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.880686    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"]
Nov 02 17:57:49.881112 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.881058    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" containerID="cri-o://b1fa5362a20ea725bfa8f1ffa2cca41316204b5e916f06785b1211f11ff89a7e" gracePeriod=30
Nov 02 17:57:49.883746 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.883725    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"]
Nov 02 17:57:49.883927 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:49.883902    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" containerID="cri-o://b88735322686996e0f53179b10a8dcd1d90d2043337590b13ffcb7a1709df640" gracePeriod=30
Nov 02 17:57:50.073639 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.073602    4956 generic.go:356] "Generic (PLEG): container finished" podID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerID="87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0" exitCode=0
Nov 02 17:57:50.073822 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.073655    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"
Nov 02 17:57:50.073822 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.073688    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" event={"ID":"e1abc52b-f034-464d-87d5-ceb44b55255d","Type":"ContainerDied","Data":"87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0"}
Nov 02 17:57:50.073822 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.073745    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww" event={"ID":"e1abc52b-f034-464d-87d5-ceb44b55255d","Type":"ContainerDied","Data":"add5a00d255e9debdc249c17aaad38ec06cdf181d3b0b75f5ea707090ce8f6d8"}
Nov 02 17:57:50.073822 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.073770    4956 scope.go:122] "RemoveContainer" containerID="87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0"
Nov 02 17:57:50.083383 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.083365    4956 scope.go:122] "RemoveContainer" containerID="87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0"
Nov 02 17:57:50.083663 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:57:50.083642    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0\": container with ID starting with 87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0 not found: ID does not exist" containerID="87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0"
Nov 02 17:57:50.083741 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.083672    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0"} err="failed to get container status \"87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0\": rpc error: code = NotFound desc = could not find container \"87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0\": container with ID starting with 87604ad67576ffe845077c8532d4994aa3e4655ec7cf97bcc63ff6c8c2dbd9f0 not found: ID does not exist"
Nov 02 17:57:50.130566 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.130431    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:57:50.131500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.131480    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 17:57:50.229479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.229447    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"]
Nov 02 17:57:50.233721 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:50.233693    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-t8hww"]
Nov 02 17:57:51.471034 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:51.470997    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:57:51.471527 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:51.471071    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:57:51.802843 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:51.802761    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:57:51.802843 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:51.802829    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:57:51.865816 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:51.865784    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" path="/var/lib/kubelet/pods/e1abc52b-f034-464d-87d5-ceb44b55255d/volumes"
Nov 02 17:57:55.112599 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.112563    4956 generic.go:356] "Generic (PLEG): container finished" podID="c844b2aa-bc8f-4996-af46-41669acac641" containerID="b88735322686996e0f53179b10a8dcd1d90d2043337590b13ffcb7a1709df640" exitCode=0
Nov 02 17:57:55.113065 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.112694    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" event={"ID":"c844b2aa-bc8f-4996-af46-41669acac641","Type":"ContainerDied","Data":"b88735322686996e0f53179b10a8dcd1d90d2043337590b13ffcb7a1709df640"}
Nov 02 17:57:55.114832 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.114807    4956 generic.go:356] "Generic (PLEG): container finished" podID="cade3928-f203-4e74-8998-d541afacb9a5" containerID="b1fa5362a20ea725bfa8f1ffa2cca41316204b5e916f06785b1211f11ff89a7e" exitCode=0
Nov 02 17:57:55.114962 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.114901    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" event={"ID":"cade3928-f203-4e74-8998-d541afacb9a5","Type":"ContainerDied","Data":"b1fa5362a20ea725bfa8f1ffa2cca41316204b5e916f06785b1211f11ff89a7e"}
Nov 02 17:57:55.164915 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.164851    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:57:55.193454 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.193432    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:57:55.232313 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232281    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-envoy\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.232500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232347    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cade3928-f203-4e74-8998-d541afacb9a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cade3928-f203-4e74-8998-d541afacb9a5-istio-podinfo\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.232500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232391    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-kube-api-access-7db6x\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-kube-api-access-7db6x\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.232500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232449    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-data\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.232500 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232476    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c844b2aa-bc8f-4996-af46-41669acac641-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844b2aa-bc8f-4996-af46-41669acac641-istio-podinfo\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.232726 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232507    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-credential-socket\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.232726 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232582    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-istio-token\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-istio-token\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.232726 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232620    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cade3928-f203-4e74-8998-d541afacb9a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cade3928-f203-4e74-8998-d541afacb9a5-istiod-ca-cert\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.232726 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232698    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-socket\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.232987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232737    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-kube-api-access-5w7cq\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-kube-api-access-5w7cq\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.232987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232774    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-socket\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.232987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232824    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-istio-token\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-istio-token\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.232987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232877    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-credential-socket\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.232987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232943    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-certs\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.232987 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.232978    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c844b2aa-bc8f-4996-af46-41669acac641-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844b2aa-bc8f-4996-af46-41669acac641-istiod-ca-cert\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.233351 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.233018    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-data\") pod \"c844b2aa-bc8f-4996-af46-41669acac641\" (UID: \"c844b2aa-bc8f-4996-af46-41669acac641\") "
Nov 02 17:57:55.233351 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.233015    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-data" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.233351 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.233058    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-envoy\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.233351 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.233115    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-certs\") pod \"cade3928-f203-4e74-8998-d541afacb9a5\" (UID: \"cade3928-f203-4e74-8998-d541afacb9a5\") "
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.233642    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-certs" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.233732    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.233981    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-data" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.234316    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c844b2aa-bc8f-4996-af46-41669acac641-istiod-ca-cert" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.234366    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-credential-socket" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.234567    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-socket" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.234577    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-certs" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.234775    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-socket" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.234818 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.234787    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-credential-socket" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.235625 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.235141    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cade3928-f203-4e74-8998-d541afacb9a5-istiod-ca-cert" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:55.236212 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.236184    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-envoy" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.236691 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.236662    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-kube-api-access-7db6x" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "kube-api-access-7db6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:55.237032 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.237006    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-envoy" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:55.237165 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.237012    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cade3928-f203-4e74-8998-d541afacb9a5-istio-podinfo" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:55.237165 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.237091    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c844b2aa-bc8f-4996-af46-41669acac641-istio-podinfo" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:55.237277 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.237213    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-kube-api-access-5w7cq" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "kube-api-access-5w7cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:55.237314 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.237271    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-istio-token" pod "c844b2aa-bc8f-4996-af46-41669acac641" (UID: "c844b2aa-bc8f-4996-af46-41669acac641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:55.238781 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.238762    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-istio-token" pod "cade3928-f203-4e74-8998-d541afacb9a5" (UID: "cade3928-f203-4e74-8998-d541afacb9a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:55.334803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334765    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.334803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334793    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.334803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334802    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844b2aa-bc8f-4996-af46-41669acac641-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.334803 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334811    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334820    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334828    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334836    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334844    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cade3928-f203-4e74-8998-d541afacb9a5-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334852    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7db6x\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-kube-api-access-7db6x\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334862    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844b2aa-bc8f-4996-af46-41669acac641-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334871    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334878    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cade3928-f203-4e74-8998-d541afacb9a5-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334888    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cade3928-f203-4e74-8998-d541afacb9a5-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334897    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cade3928-f203-4e74-8998-d541afacb9a5-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334904    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7cq\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-kube-api-access-5w7cq\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334912    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844b2aa-bc8f-4996-af46-41669acac641-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:55.335167 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:55.334920    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844b2aa-bc8f-4996-af46-41669acac641-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:56.122042 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.122008    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"
Nov 02 17:57:56.122042 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.122026    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct" event={"ID":"c844b2aa-bc8f-4996-af46-41669acac641","Type":"ContainerDied","Data":"7399f75897c781a277911042e25f55bf2f22d8c0fe709e376c88f36d4abaee77"}
Nov 02 17:57:56.122595 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.122094    4956 scope.go:122] "RemoveContainer" containerID="b88735322686996e0f53179b10a8dcd1d90d2043337590b13ffcb7a1709df640"
Nov 02 17:57:56.123684 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.123666    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"
Nov 02 17:57:56.123758 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.123690    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s" event={"ID":"cade3928-f203-4e74-8998-d541afacb9a5","Type":"ContainerDied","Data":"5b681ed9b0b2d735bff546b3788f58240662f9f0081e43f1fd06cbda8f9ef42a"}
Nov 02 17:57:56.132070 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.132058    4956 scope.go:122] "RemoveContainer" containerID="b1fa5362a20ea725bfa8f1ffa2cca41316204b5e916f06785b1211f11ff89a7e"
Nov 02 17:57:56.338229 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.338187    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"]
Nov 02 17:57:56.387947 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.387872    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-hhc8s"]
Nov 02 17:57:56.453312 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.453279    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"]
Nov 02 17:57:56.460919 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:56.460876    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-7jhct"]
Nov 02 17:57:57.865136 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:57.865096    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c844b2aa-bc8f-4996-af46-41669acac641" path="/var/lib/kubelet/pods/c844b2aa-bc8f-4996-af46-41669acac641/volumes"
Nov 02 17:57:57.865593 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:57.865578    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cade3928-f203-4e74-8998-d541afacb9a5" path="/var/lib/kubelet/pods/cade3928-f203-4e74-8998-d541afacb9a5/volumes"
Nov 02 17:57:59.064545 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:57:59.064504    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:57:59.433851 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:59.433775    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"]
Nov 02 17:57:59.433997 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:57:59.433976    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" containerID="cri-o://4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739" gracePeriod=30
Nov 02 17:58:03.970573 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:03.970535    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:58:03.970970 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:03.970606    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:58:04.739744 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.739720    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:58:04.853341 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853308    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827b7174-42fe-4f11-a072-83842c49f869-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827b7174-42fe-4f11-a072-83842c49f869-istiod-ca-cert\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853383    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-istio-token\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-istio-token\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853440    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-certs\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853505    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-socket\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853551 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853532    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-credential-socket\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853579    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-data\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853670    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/827b7174-42fe-4f11-a072-83842c49f869-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827b7174-42fe-4f11-a072-83842c49f869-istio-podinfo\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853783 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853711    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-kube-api-access-nr4hl\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-kube-api-access-nr4hl\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853932 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853807    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-envoy\") pod \"827b7174-42fe-4f11-a072-83842c49f869\" (UID: \"827b7174-42fe-4f11-a072-83842c49f869\") "
Nov 02 17:58:04.853932 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853832    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-certs" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.853932 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853845    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-socket" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.853932 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853864    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827b7174-42fe-4f11-a072-83842c49f869-istiod-ca-cert" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:04.853932 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.853870    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-credential-socket" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.854168 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.854111    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-data" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.854383 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.854362    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827b7174-42fe-4f11-a072-83842c49f869-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.854383 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.854389    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.854383 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.854410    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.854383 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.854425    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.854789 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.854439    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.855868 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.855844    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-istio-token" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.856027 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.856002    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-kube-api-access-nr4hl" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "kube-api-access-nr4hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.856097 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.856080    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/827b7174-42fe-4f11-a072-83842c49f869-istio-podinfo" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:58:04.856288 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.856263    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-envoy" pod "827b7174-42fe-4f11-a072-83842c49f869" (UID: "827b7174-42fe-4f11-a072-83842c49f869"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.955715 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.955677    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.955922 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.955905    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827b7174-42fe-4f11-a072-83842c49f869-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.956050 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.956023    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr4hl\" (UniqueName: \"kubernetes.io/projected/827b7174-42fe-4f11-a072-83842c49f869-kube-api-access-nr4hl\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.956050 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:04.956049    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827b7174-42fe-4f11-a072-83842c49f869-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:05.192547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.192451    4956 generic.go:356] "Generic (PLEG): container finished" podID="827b7174-42fe-4f11-a072-83842c49f869" containerID="4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739" exitCode=0
Nov 02 17:58:05.192547 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.192508    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"
Nov 02 17:58:05.193052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.192505    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" event={"ID":"827b7174-42fe-4f11-a072-83842c49f869","Type":"ContainerDied","Data":"4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739"}
Nov 02 17:58:05.193052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.192652    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg" event={"ID":"827b7174-42fe-4f11-a072-83842c49f869","Type":"ContainerDied","Data":"fb1553f3d47ca74b66ca7d9861c3400fdf5914397f17ace26dea9fdde9e9e63f"}
Nov 02 17:58:05.193052 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.192694    4956 scope.go:122] "RemoveContainer" containerID="4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739"
Nov 02 17:58:05.204042 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.204018    4956 scope.go:122] "RemoveContainer" containerID="4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739"
Nov 02 17:58:05.204371 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:58:05.204348    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739\": container with ID starting with 4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739 not found: ID does not exist" containerID="4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739"
Nov 02 17:58:05.204458 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.204386    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739"} err="failed to get container status \"4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739\": rpc error: code = NotFound desc = could not find container \"4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739\": container with ID starting with 4945d45a7cd51cce895b67023ec98e0d0cef76dc1dd2f80839f9dd0dfe3fa739 not found: ID does not exist"
Nov 02 17:58:05.361141 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.361081    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"]
Nov 02 17:58:05.410239 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.410192    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4g6dg"]
Nov 02 17:58:05.865007 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:05.864970    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827b7174-42fe-4f11-a072-83842c49f869" path="/var/lib/kubelet/pods/827b7174-42fe-4f11-a072-83842c49f869/volumes"
Nov 02 17:58:18.286179 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:18.286146    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"]
Nov 02 17:58:18.286596 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:18.286348    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" containerID="cri-o://56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60" gracePeriod=30
Nov 02 17:58:20.723642 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:20.723593    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:58:20.724111 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:20.723675    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:58:23.553461 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.553402    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:58:23.694037 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694006    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-socket\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694037 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694042    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-podinfo\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694083    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-kube-api-access-z5vsf\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-kube-api-access-z5vsf\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694109    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istiod-ca-cert\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694156    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-data\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694316 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694240    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-credential-socket\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694365    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-certs\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694360    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-socket" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.694524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694471    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-token\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-token\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694524 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694505    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-envoy\") pod \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\" (UID: \"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e\") "
Nov 02 17:58:23.694724 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694562    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-credential-socket" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.694724 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694582    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-certs" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.694724 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694696    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-data" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.694873 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.694810    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istiod-ca-cert" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:23.695270 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.695251    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.695359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.695277    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.695359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.695295    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.695359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.695311    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.695359 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.695326    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.696706 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.696685    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-envoy" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.696836 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.696811    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-podinfo" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:58:23.697162 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.697114    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-token" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:23.697162 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.697141    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-kube-api-access-z5vsf" pod "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" (UID: "fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e"). InnerVolumeSpecName "kube-api-access-z5vsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:23.796764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.796728    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.796764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.796757    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.796764 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.796768    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.797003 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:23.796777    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vsf\" (UniqueName: \"kubernetes.io/projected/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e-kube-api-access-z5vsf\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:24.319166 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.319099    4956 generic.go:356] "Generic (PLEG): container finished" podID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerID="56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60" exitCode=0
Nov 02 17:58:24.319385 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.319347    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" event={"ID":"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e","Type":"ContainerDied","Data":"56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60"}
Nov 02 17:58:24.319458 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.319439    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9" event={"ID":"fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e","Type":"ContainerDied","Data":"022c357fb21689aea45830c570ce0c984d69fd9800cdeab35abab23e3bae0de0"}
Nov 02 17:58:24.319510 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.319486    4956 scope.go:122] "RemoveContainer" containerID="56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60"
Nov 02 17:58:24.319850 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.319697    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"
Nov 02 17:58:24.332100 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.332074    4956 scope.go:122] "RemoveContainer" containerID="56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60"
Nov 02 17:58:24.332421 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:58:24.332390    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60\": container with ID starting with 56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60 not found: ID does not exist" containerID="56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60"
Nov 02 17:58:24.332505 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.332424    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60"} err="failed to get container status \"56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60\": rpc error: code = NotFound desc = could not find container \"56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60\": container with ID starting with 56d895621c1eaea6a6e600440cc20774406c01984512cf25baa9e4a993eb5a60 not found: ID does not exist"
Nov 02 17:58:24.494473 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.494427    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"]
Nov 02 17:58:24.497923 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:24.497895    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-spqk9"]
Nov 02 17:58:25.864744 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:25.864705    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" path="/var/lib/kubelet/pods/fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e/volumes"
Nov 02 17:58:29.054785 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:58:29.054746    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:58:55.705479 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.705445    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"]
Nov 02 17:58:55.705989 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.705974    4956 cpu_manager.go:414] "containerMap: removing container" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy"
Nov 02 17:58:55.706044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.705991    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy"
Nov 02 17:58:55.706044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706003    4956 cpu_manager.go:414] "containerMap: removing container" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy"
Nov 02 17:58:55.706044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706008    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy"
Nov 02 17:58:55.706044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706030    4956 cpu_manager.go:414] "containerMap: removing container" podUID="63c7f52c-7708-4c5f-97b8-029e426f9c0f" containerName="payload-pre-processing"
Nov 02 17:58:55.706044 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706035    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c7f52c-7708-4c5f-97b8-029e426f9c0f" containerName="payload-pre-processing"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706052    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706057    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706064    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706070    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706095    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706099    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706214    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="827b7174-42fe-4f11-a072-83842c49f869" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706224    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="cade3928-f203-4e74-8998-d541afacb9a5" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706229    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c844b2aa-bc8f-4996-af46-41669acac641" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706238    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad8d91f-6ce0-4f10-a4fa-f67a5dd25a4e" containerName="istio-proxy"
Nov 02 17:58:55.706245 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706245    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1abc52b-f034-464d-87d5-ceb44b55255d" containerName="istio-proxy"
Nov 02 17:58:55.706611 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.706251    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c7f52c-7708-4c5f-97b8-029e426f9c0f" containerName="payload-pre-processing"
Nov 02 17:58:55.710956 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.710936    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:55.766683 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.766642    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"]
Nov 02 17:58:55.812394 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.812359    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f42381\""
Nov 02 17:58:55.870265 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.870230    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1776362a-6635-47cf-b50f-589d5bd5c900-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:55.870492 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.870290    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpl7\" (UniqueName: \"kubernetes.io/projected/1776362a-6635-47cf-b50f-589d5bd5c900-kube-api-access-5xpl7\") pod \"payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:55.971771 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.971670    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1776362a-6635-47cf-b50f-589d5bd5c900-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:55.971771 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.971717    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpl7\" (UniqueName: \"kubernetes.io/projected/1776362a-6635-47cf-b50f-589d5bd5c900-kube-api-access-5xpl7\") pod \"payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:55.972384 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:55.972361    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1776362a-6635-47cf-b50f-589d5bd5c900-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:56.028916 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:56.028881    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpl7\" (UniqueName: \"kubernetes.io/projected/1776362a-6635-47cf-b50f-589d5bd5c900-kube-api-access-5xpl7\") pod \"payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:56.324426 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:56.324341    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:56.702559 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:56.702526    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"]
Nov 02 17:58:56.703706 ip-10-0-129-72 kubenswrapper[4956]: W1102 17:58:56.703681    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1776362a_6635_47cf_b50f_589d5bd5c900.slice/crio-fb8f18e359fa360c62f48715538659a8a4f817586d853a8132e3af942f319ae8 WatchSource:0}: Error finding container fb8f18e359fa360c62f48715538659a8a4f817586d853a8132e3af942f319ae8: Status 404 returned error can't find the container with id fb8f18e359fa360c62f48715538659a8a4f817586d853a8132e3af942f319ae8
Nov 02 17:58:57.510012 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:57.509978    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn" event={"ID":"1776362a-6635-47cf-b50f-589d5bd5c900","Type":"ContainerStarted","Data":"412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23"}
Nov 02 17:58:57.510572 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:57.510045    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn" event={"ID":"1776362a-6635-47cf-b50f-589d5bd5c900","Type":"ContainerStarted","Data":"fb8f18e359fa360c62f48715538659a8a4f817586d853a8132e3af942f319ae8"}
Nov 02 17:58:57.510572 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:57.510214    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:58:57.632299 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:58:57.632250    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn" podStartSLOduration=2.632235689 podStartE2EDuration="2.632235689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:57.628993892 +0000 UTC m=+2167.886024511" watchObservedRunningTime="2026-11-02 17:58:57.632235689 +0000 UTC m=+2167.889266307"
Nov 02 17:58:59.052343 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:58:59.052291    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:59:08.519961 ip-10-0-129-72 kubenswrapper[4956]: I1102 17:59:08.519926    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 17:59:29.055635 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:59:29.055595    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 17:59:59.052592 ip-10-0-129-72 kubenswrapper[4956]: E1102 17:59:59.052531    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:00:00.173790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.173754    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"]
Nov 02 18:00:00.177781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.177760    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.183953 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.183911    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"]
Nov 02 18:00:00.188335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.188312    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.203321 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.203285    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"]
Nov 02 18:00:00.207392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.207365    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.223225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.223181    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"]
Nov 02 18:00:00.227165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.227095    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.234227 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.234173    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"]
Nov 02 18:00:00.238773 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.238749    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.259423 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259360    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e580309-6fbb-48e4-9f1e-93cba517df80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.259606 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259489    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knrc\" (UniqueName: \"kubernetes.io/projected/725c3153-c34e-49e8-ada1-af6d759c6d1e-kube-api-access-2knrc\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.259656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259639    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd2n4\" (UniqueName: \"kubernetes.io/projected/5e580309-6fbb-48e4-9f1e-93cba517df80-kube-api-access-cd2n4\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.259719 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259704    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f8vb\" (UniqueName: \"kubernetes.io/projected/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-kube-api-access-6f8vb\") pod \"maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.259755 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259735    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41479ffc-a4e0-40f4-a6db-72efe1e31c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-g9qqx\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.259836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259821    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c3153-c34e-49e8-ada1-af6d759c6d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.259921 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259908    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.260012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.259946    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2g45\" (UniqueName: \"kubernetes.io/projected/41479ffc-a4e0-40f4-a6db-72efe1e31c10-kube-api-access-n2g45\") pod \"maas-api-key-cleanup-29894040-g9qqx\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.272057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.272018    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"]
Nov 02 18:00:00.275000 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.274974    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"]
Nov 02 18:00:00.324524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.324480    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"]
Nov 02 18:00:00.327136 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.327088    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"]
Nov 02 18:00:00.329840 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.329816    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:00:00.361071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361027    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f8vb\" (UniqueName: \"kubernetes.io/projected/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-kube-api-access-6f8vb\") pod \"maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.361071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361071    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41479ffc-a4e0-40f4-a6db-72efe1e31c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-g9qqx\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.361372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361250    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c3153-c34e-49e8-ada1-af6d759c6d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.361372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361334    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqxp\" (UniqueName: \"kubernetes.io/projected/07ef0ef6-7486-400d-a613-7f5aaa389aae-kube-api-access-2hqxp\") pod \"maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.361494 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361371    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.361494 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361411    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2g45\" (UniqueName: \"kubernetes.io/projected/41479ffc-a4e0-40f4-a6db-72efe1e31c10-kube-api-access-n2g45\") pod \"maas-api-key-cleanup-29894040-g9qqx\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.361494 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361475    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef0ef6-7486-400d-a613-7f5aaa389aae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.361659 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361517    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e580309-6fbb-48e4-9f1e-93cba517df80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.361659 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361553    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knrc\" (UniqueName: \"kubernetes.io/projected/725c3153-c34e-49e8-ada1-af6d759c6d1e-kube-api-access-2knrc\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.361782 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361762    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd2n4\" (UniqueName: \"kubernetes.io/projected/5e580309-6fbb-48e4-9f1e-93cba517df80-kube-api-access-cd2n4\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.361922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.361900    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41479ffc-a4e0-40f4-a6db-72efe1e31c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-g9qqx\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.362195 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.362138    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.362657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.362636    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c3153-c34e-49e8-ada1-af6d759c6d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.372605 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.372567    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e580309-6fbb-48e4-9f1e-93cba517df80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.375867 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.375833    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"]
Nov 02 18:00:00.419784 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.419746    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f8vb\" (UniqueName: \"kubernetes.io/projected/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-kube-api-access-6f8vb\") pod \"maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.419938 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.419753    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2g45\" (UniqueName: \"kubernetes.io/projected/41479ffc-a4e0-40f4-a6db-72efe1e31c10-kube-api-access-n2g45\") pod \"maas-api-key-cleanup-29894040-g9qqx\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.427993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.427914    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd2n4\" (UniqueName: \"kubernetes.io/projected/5e580309-6fbb-48e4-9f1e-93cba517df80-kube-api-access-cd2n4\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.428163 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.428054    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knrc\" (UniqueName: \"kubernetes.io/projected/725c3153-c34e-49e8-ada1-af6d759c6d1e-kube-api-access-2knrc\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.462690 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.462649    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqxp\" (UniqueName: \"kubernetes.io/projected/07ef0ef6-7486-400d-a613-7f5aaa389aae-kube-api-access-2hqxp\") pod \"maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.462890 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.462708    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef0ef6-7486-400d-a613-7f5aaa389aae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.463415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.463392    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef0ef6-7486-400d-a613-7f5aaa389aae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.492940 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.492902    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:00.502545 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.502515    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:00.523204 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.522717    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:00.523558 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.523415    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqxp\" (UniqueName: \"kubernetes.io/projected/07ef0ef6-7486-400d-a613-7f5aaa389aae-kube-api-access-2hqxp\") pod \"maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.543147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.543073    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:00.554387 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.554347    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:00.893074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.893043    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"]
Nov 02 18:00:00.893975 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:00:00.893949    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725c3153_c34e_49e8_ada1_af6d759c6d1e.slice/crio-085ed0f8ede10efe082f5b7b201323a796aea501f57e3a5eba3477600cbd65bd WatchSource:0}: Error finding container 085ed0f8ede10efe082f5b7b201323a796aea501f57e3a5eba3477600cbd65bd: Status 404 returned error can't find the container with id 085ed0f8ede10efe082f5b7b201323a796aea501f57e3a5eba3477600cbd65bd
Nov 02 18:00:00.895907 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.895887    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:00:00.944673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.944627    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"]
Nov 02 18:00:00.946064 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:00:00.946034    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e580309_6fbb_48e4_9f1e_93cba517df80.slice/crio-b9affb1f2356bfd81fe8e08b94efa84d8b8fb1774c26308cfba4e64e6e6eead1 WatchSource:0}: Error finding container b9affb1f2356bfd81fe8e08b94efa84d8b8fb1774c26308cfba4e64e6e6eead1: Status 404 returned error can't find the container with id b9affb1f2356bfd81fe8e08b94efa84d8b8fb1774c26308cfba4e64e6e6eead1
Nov 02 18:00:00.947485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.947458    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"]
Nov 02 18:00:00.948680 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:00:00.948658    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e14e8cc_e7a0_4701_8ac2_7c3ff57e95ac.slice/crio-cb3c7ac5804bae296648944937e8175471cd4aa91944b2db56c6093d94496e95 WatchSource:0}: Error finding container cb3c7ac5804bae296648944937e8175471cd4aa91944b2db56c6093d94496e95: Status 404 returned error can't find the container with id cb3c7ac5804bae296648944937e8175471cd4aa91944b2db56c6093d94496e95
Nov 02 18:00:00.999916 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:00.999894    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"]
Nov 02 18:00:01.005579 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:01.005556    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"]
Nov 02 18:00:01.006896 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:00:01.006869    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ef0ef6_7486_400d_a613_7f5aaa389aae.slice/crio-d24af16876dbf139ace7ad009437bf32b8da37f247f8ad1f50d8e8f1cb8d3075 WatchSource:0}: Error finding container d24af16876dbf139ace7ad009437bf32b8da37f247f8ad1f50d8e8f1cb8d3075: Status 404 returned error can't find the container with id d24af16876dbf139ace7ad009437bf32b8da37f247f8ad1f50d8e8f1cb8d3075
Nov 02 18:00:01.882522 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:01.882486    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" event={"ID":"725c3153-c34e-49e8-ada1-af6d759c6d1e","Type":"ContainerStarted","Data":"085ed0f8ede10efe082f5b7b201323a796aea501f57e3a5eba3477600cbd65bd"}
Nov 02 18:00:01.883925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:01.883891    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" event={"ID":"07ef0ef6-7486-400d-a613-7f5aaa389aae","Type":"ContainerStarted","Data":"d24af16876dbf139ace7ad009437bf32b8da37f247f8ad1f50d8e8f1cb8d3075"}
Nov 02 18:00:01.885466 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:01.885441    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" event={"ID":"5e580309-6fbb-48e4-9f1e-93cba517df80","Type":"ContainerStarted","Data":"b9affb1f2356bfd81fe8e08b94efa84d8b8fb1774c26308cfba4e64e6e6eead1"}
Nov 02 18:00:01.886932 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:01.886913    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" event={"ID":"41479ffc-a4e0-40f4-a6db-72efe1e31c10","Type":"ContainerStarted","Data":"447d5b6b0aa9961c73e5a5f77ab042521cad05a3665bb3b658170cd7cc6befc1"}
Nov 02 18:00:01.888107 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:01.888090    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" event={"ID":"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac","Type":"ContainerStarted","Data":"cb3c7ac5804bae296648944937e8175471cd4aa91944b2db56c6093d94496e95"}
Nov 02 18:00:06.923187 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:06.923073    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" event={"ID":"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac","Type":"ContainerStarted","Data":"7c0261eea9a8a0a5786842b52ffbf791a102e140a28a856173cf7468eb7a0362"}
Nov 02 18:00:07.038338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:07.038280    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" podStartSLOduration=1.778774131 podStartE2EDuration="7.03826551s" totalImagesPullingTime="5.259491379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.037520783 +0000 UTC m=+2237.294551407" watchObservedRunningTime="2026-11-02 18:00:07.03826551 +0000 UTC m=+2237.295296129"
Nov 02 18:00:07.932933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:07.932839    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" event={"ID":"07ef0ef6-7486-400d-a613-7f5aaa389aae","Type":"ContainerStarted","Data":"32cbe791e74e502acfeadde36ef91ae888d0197763ca67341455bea23fc4e88f"}
Nov 02 18:00:07.934592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:07.934557    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" event={"ID":"5e580309-6fbb-48e4-9f1e-93cba517df80","Type":"ContainerStarted","Data":"d12d32fb4a576a2ac2a502834c7f47baeb2b2adf56d2e040fd4d893bd4fe6ac2"}
Nov 02 18:00:07.936030 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:07.936008    4956 generic.go:356] "Generic (PLEG): container finished" podID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerID="7c0261eea9a8a0a5786842b52ffbf791a102e140a28a856173cf7468eb7a0362" exitCode=60
Nov 02 18:00:07.936144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:07.936083    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" event={"ID":"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac","Type":"ContainerDied","Data":"7c0261eea9a8a0a5786842b52ffbf791a102e140a28a856173cf7468eb7a0362"}
Nov 02 18:00:07.936273 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:07.936257    4956 scope.go:122] "RemoveContainer" containerID="7c0261eea9a8a0a5786842b52ffbf791a102e140a28a856173cf7468eb7a0362"
Nov 02 18:00:07.937629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:07.937606    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" event={"ID":"725c3153-c34e-49e8-ada1-af6d759c6d1e","Type":"ContainerStarted","Data":"12ac96376f9f6e8846b0d06be9b6e5561597e8075a192e862ef4e4a8e1bdc860"}
Nov 02 18:00:08.074058 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.074006    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" podStartSLOduration=1.7088743910000002 podStartE2EDuration="8.073991067s" totalImagesPullingTime="6.365116676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:08.070230521 +0000 UTC m=+2238.327261140" watchObservedRunningTime="2026-11-02 18:00:08.073991067 +0000 UTC m=+2238.331021686"
Nov 02 18:00:08.185349 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.185231    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" podStartSLOduration=1.668853972 podStartE2EDuration="8.185212639s" totalImagesPullingTime="6.516358667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:08.18462426 +0000 UTC m=+2238.441654883" watchObservedRunningTime="2026-11-02 18:00:08.185212639 +0000 UTC m=+2238.442243260"
Nov 02 18:00:08.300644 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.300584    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" podStartSLOduration=1.874739355 podStartE2EDuration="8.300569344s" totalImagesPullingTime="6.425829989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:08.299239256 +0000 UTC m=+2238.556269876" watchObservedRunningTime="2026-11-02 18:00:08.300569344 +0000 UTC m=+2238.557599964"
Nov 02 18:00:08.945072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.945037    4956 generic.go:356] "Generic (PLEG): container finished" podID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerID="12ac96376f9f6e8846b0d06be9b6e5561597e8075a192e862ef4e4a8e1bdc860" exitCode=60
Nov 02 18:00:08.945639 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.945153    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" event={"ID":"725c3153-c34e-49e8-ada1-af6d759c6d1e","Type":"ContainerDied","Data":"12ac96376f9f6e8846b0d06be9b6e5561597e8075a192e862ef4e4a8e1bdc860"}
Nov 02 18:00:08.945639 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.945493    4956 scope.go:122] "RemoveContainer" containerID="12ac96376f9f6e8846b0d06be9b6e5561597e8075a192e862ef4e4a8e1bdc860"
Nov 02 18:00:08.947011 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.946986    4956 generic.go:356] "Generic (PLEG): container finished" podID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerID="32cbe791e74e502acfeadde36ef91ae888d0197763ca67341455bea23fc4e88f" exitCode=60
Nov 02 18:00:08.947148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.947071    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" event={"ID":"07ef0ef6-7486-400d-a613-7f5aaa389aae","Type":"ContainerDied","Data":"32cbe791e74e502acfeadde36ef91ae888d0197763ca67341455bea23fc4e88f"}
Nov 02 18:00:08.947339 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.947326    4956 scope.go:122] "RemoveContainer" containerID="32cbe791e74e502acfeadde36ef91ae888d0197763ca67341455bea23fc4e88f"
Nov 02 18:00:08.950073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.950053    4956 generic.go:356] "Generic (PLEG): container finished" podID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerID="d12d32fb4a576a2ac2a502834c7f47baeb2b2adf56d2e040fd4d893bd4fe6ac2" exitCode=60
Nov 02 18:00:08.950199 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.950158    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" event={"ID":"5e580309-6fbb-48e4-9f1e-93cba517df80","Type":"ContainerDied","Data":"d12d32fb4a576a2ac2a502834c7f47baeb2b2adf56d2e040fd4d893bd4fe6ac2"}
Nov 02 18:00:08.950412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.950392    4956 scope.go:122] "RemoveContainer" containerID="d12d32fb4a576a2ac2a502834c7f47baeb2b2adf56d2e040fd4d893bd4fe6ac2"
Nov 02 18:00:08.953574 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.953555    4956 generic.go:356] "Generic (PLEG): container finished" podID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerID="cb34cd80b23dc10371070e2d2137dd68641f6d117edd6fc9b6f50b8ef0267e46" exitCode=60
Nov 02 18:00:08.953664 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.953595    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" event={"ID":"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac","Type":"ContainerDied","Data":"cb34cd80b23dc10371070e2d2137dd68641f6d117edd6fc9b6f50b8ef0267e46"}
Nov 02 18:00:08.953664 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.953642    4956 scope.go:122] "RemoveContainer" containerID="7c0261eea9a8a0a5786842b52ffbf791a102e140a28a856173cf7468eb7a0362"
Nov 02 18:00:08.953914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:08.953894    4956 scope.go:122] "RemoveContainer" containerID="cb34cd80b23dc10371070e2d2137dd68641f6d117edd6fc9b6f50b8ef0267e46"
Nov 02 18:00:08.954207 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:08.954173    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm_odh-ai-gateway-infra(4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac"
Nov 02 18:00:09.970477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.970435    4956 generic.go:356] "Generic (PLEG): container finished" podID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerID="1637434ae089aa1e90755c7b034e9a98c72ab8b2d638baa98d17d11865162473" exitCode=60
Nov 02 18:00:09.970931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.970515    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" event={"ID":"725c3153-c34e-49e8-ada1-af6d759c6d1e","Type":"ContainerDied","Data":"1637434ae089aa1e90755c7b034e9a98c72ab8b2d638baa98d17d11865162473"}
Nov 02 18:00:09.970931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.970592    4956 scope.go:122] "RemoveContainer" containerID="12ac96376f9f6e8846b0d06be9b6e5561597e8075a192e862ef4e4a8e1bdc860"
Nov 02 18:00:09.970931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.970871    4956 scope.go:122] "RemoveContainer" containerID="1637434ae089aa1e90755c7b034e9a98c72ab8b2d638baa98d17d11865162473"
Nov 02 18:00:09.971212 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:09.971191    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw_odh-ai-gateway-infra(725c3153-c34e-49e8-ada1-af6d759c6d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e"
Nov 02 18:00:09.972438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.972419    4956 generic.go:356] "Generic (PLEG): container finished" podID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerID="6899fb0e18e678bea2ce61764bf9a690b03102f575f570caf34d9f7a1799d033" exitCode=60
Nov 02 18:00:09.972531 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.972501    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" event={"ID":"07ef0ef6-7486-400d-a613-7f5aaa389aae","Type":"ContainerDied","Data":"6899fb0e18e678bea2ce61764bf9a690b03102f575f570caf34d9f7a1799d033"}
Nov 02 18:00:09.972793 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.972776    4956 scope.go:122] "RemoveContainer" containerID="6899fb0e18e678bea2ce61764bf9a690b03102f575f570caf34d9f7a1799d033"
Nov 02 18:00:09.973042 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:09.973022    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4_odh-ai-gateway-infra(07ef0ef6-7486-400d-a613-7f5aaa389aae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae"
Nov 02 18:00:09.974329 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.974311    4956 generic.go:356] "Generic (PLEG): container finished" podID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerID="f507d6dc91e8ce0b28cf8878ed7f4739dc641fb3123b8a2d939e83d07f315c52" exitCode=60
Nov 02 18:00:09.974410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.974390    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" event={"ID":"5e580309-6fbb-48e4-9f1e-93cba517df80","Type":"ContainerDied","Data":"f507d6dc91e8ce0b28cf8878ed7f4739dc641fb3123b8a2d939e83d07f315c52"}
Nov 02 18:00:09.974675 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.974662    4956 scope.go:122] "RemoveContainer" containerID="f507d6dc91e8ce0b28cf8878ed7f4739dc641fb3123b8a2d939e83d07f315c52"
Nov 02 18:00:09.974929 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:09.974911    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr_odh-ai-gateway-infra(5e580309-6fbb-48e4-9f1e-93cba517df80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80"
Nov 02 18:00:09.976283 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:09.976259    4956 scope.go:122] "RemoveContainer" containerID="cb34cd80b23dc10371070e2d2137dd68641f6d117edd6fc9b6f50b8ef0267e46"
Nov 02 18:00:09.976616 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:09.976593    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm_odh-ai-gateway-infra(4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac"
Nov 02 18:00:10.100621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:10.100599    4956 scope.go:122] "RemoveContainer" containerID="32cbe791e74e502acfeadde36ef91ae888d0197763ca67341455bea23fc4e88f"
Nov 02 18:00:10.113437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:10.113416    4956 scope.go:122] "RemoveContainer" containerID="d12d32fb4a576a2ac2a502834c7f47baeb2b2adf56d2e040fd4d893bd4fe6ac2"
Nov 02 18:00:10.984432 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:10.984401    4956 scope.go:122] "RemoveContainer" containerID="1637434ae089aa1e90755c7b034e9a98c72ab8b2d638baa98d17d11865162473"
Nov 02 18:00:10.984882 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:10.984754    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw_odh-ai-gateway-infra(725c3153-c34e-49e8-ada1-af6d759c6d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e"
Nov 02 18:00:10.986096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:10.986071    4956 scope.go:122] "RemoveContainer" containerID="6899fb0e18e678bea2ce61764bf9a690b03102f575f570caf34d9f7a1799d033"
Nov 02 18:00:10.986412 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:10.986376    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4_odh-ai-gateway-infra(07ef0ef6-7486-400d-a613-7f5aaa389aae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae"
Nov 02 18:00:10.987666 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:10.987650    4956 scope.go:122] "RemoveContainer" containerID="f507d6dc91e8ce0b28cf8878ed7f4739dc641fb3123b8a2d939e83d07f315c52"
Nov 02 18:00:10.987895 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:10.987878    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr_odh-ai-gateway-infra(5e580309-6fbb-48e4-9f1e-93cba517df80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80"
Nov 02 18:00:11.994990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:11.994952    4956 generic.go:356] "Generic (PLEG): container finished" podID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerID="ce60e24408c4ec9f31d737dcf6ee43560c9c4954244266ff8b36d2e18649e894" exitCode=60
Nov 02 18:00:11.995405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:11.995027    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" event={"ID":"41479ffc-a4e0-40f4-a6db-72efe1e31c10","Type":"ContainerDied","Data":"ce60e24408c4ec9f31d737dcf6ee43560c9c4954244266ff8b36d2e18649e894"}
Nov 02 18:00:11.995405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:11.995345    4956 scope.go:122] "RemoveContainer" containerID="ce60e24408c4ec9f31d737dcf6ee43560c9c4954244266ff8b36d2e18649e894"
Nov 02 18:00:13.004203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:13.004170    4956 generic.go:356] "Generic (PLEG): container finished" podID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerID="c447436d69c82028e3c4cf18a83b38869fa565ff251b171f0ec365a82ddb98e6" exitCode=60
Nov 02 18:00:13.004725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:13.004260    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" event={"ID":"41479ffc-a4e0-40f4-a6db-72efe1e31c10","Type":"ContainerDied","Data":"c447436d69c82028e3c4cf18a83b38869fa565ff251b171f0ec365a82ddb98e6"}
Nov 02 18:00:13.004725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:13.004326    4956 scope.go:122] "RemoveContainer" containerID="ce60e24408c4ec9f31d737dcf6ee43560c9c4954244266ff8b36d2e18649e894"
Nov 02 18:00:13.004725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:13.004615    4956 scope.go:122] "RemoveContainer" containerID="c447436d69c82028e3c4cf18a83b38869fa565ff251b171f0ec365a82ddb98e6"
Nov 02 18:00:13.004957 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:13.004936    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-g9qqx_odh-ai-gateway-infra(41479ffc-a4e0-40f4-a6db-72efe1e31c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10"
Nov 02 18:00:14.013295 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:14.013258    4956 scope.go:122] "RemoveContainer" containerID="c447436d69c82028e3c4cf18a83b38869fa565ff251b171f0ec365a82ddb98e6"
Nov 02 18:00:14.013709 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:14.013585    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-g9qqx_odh-ai-gateway-infra(41479ffc-a4e0-40f4-a6db-72efe1e31c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10"
Nov 02 18:00:20.858499 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:20.858467    4956 scope.go:122] "RemoveContainer" containerID="1637434ae089aa1e90755c7b034e9a98c72ab8b2d638baa98d17d11865162473"
Nov 02 18:00:21.857784 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:21.857741    4956 scope.go:122] "RemoveContainer" containerID="6899fb0e18e678bea2ce61764bf9a690b03102f575f570caf34d9f7a1799d033"
Nov 02 18:00:21.857952 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:21.857861    4956 scope.go:122] "RemoveContainer" containerID="f507d6dc91e8ce0b28cf8878ed7f4739dc641fb3123b8a2d939e83d07f315c52"
Nov 02 18:00:22.060871 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:22.060841    4956 generic.go:356] "Generic (PLEG): container finished" podID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerID="60be642017ce432586ee18c38a227a08131f4199fba871bc73eeee5022d80874" exitCode=60
Nov 02 18:00:22.061319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:22.060908    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" event={"ID":"725c3153-c34e-49e8-ada1-af6d759c6d1e","Type":"ContainerDied","Data":"60be642017ce432586ee18c38a227a08131f4199fba871bc73eeee5022d80874"}
Nov 02 18:00:22.061319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:22.060967    4956 scope.go:122] "RemoveContainer" containerID="1637434ae089aa1e90755c7b034e9a98c72ab8b2d638baa98d17d11865162473"
Nov 02 18:00:22.061422 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:22.061379    4956 scope.go:122] "RemoveContainer" containerID="60be642017ce432586ee18c38a227a08131f4199fba871bc73eeee5022d80874"
Nov 02 18:00:22.061689 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:22.061664    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw_odh-ai-gateway-infra(725c3153-c34e-49e8-ada1-af6d759c6d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e"
Nov 02 18:00:23.069944 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.069856    4956 generic.go:356] "Generic (PLEG): container finished" podID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerID="c7ac81f80102373fb62428c8a5b851b810889e2aca3d393c21ee879d2f69d812" exitCode=60
Nov 02 18:00:23.069944 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.069926    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" event={"ID":"07ef0ef6-7486-400d-a613-7f5aaa389aae","Type":"ContainerDied","Data":"c7ac81f80102373fb62428c8a5b851b810889e2aca3d393c21ee879d2f69d812"}
Nov 02 18:00:23.070483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.070020    4956 scope.go:122] "RemoveContainer" containerID="6899fb0e18e678bea2ce61764bf9a690b03102f575f570caf34d9f7a1799d033"
Nov 02 18:00:23.070483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.070450    4956 scope.go:122] "RemoveContainer" containerID="c7ac81f80102373fb62428c8a5b851b810889e2aca3d393c21ee879d2f69d812"
Nov 02 18:00:23.070802 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:23.070777    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4_odh-ai-gateway-infra(07ef0ef6-7486-400d-a613-7f5aaa389aae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae"
Nov 02 18:00:23.072113 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.072096    4956 generic.go:356] "Generic (PLEG): container finished" podID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerID="d05025dbae92f97a50d6744c62a0e3f6a358e573ff31d07bc2eb700d3753c589" exitCode=60
Nov 02 18:00:23.072212 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.072166    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" event={"ID":"5e580309-6fbb-48e4-9f1e-93cba517df80","Type":"ContainerDied","Data":"d05025dbae92f97a50d6744c62a0e3f6a358e573ff31d07bc2eb700d3753c589"}
Nov 02 18:00:23.072491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.072478    4956 scope.go:122] "RemoveContainer" containerID="d05025dbae92f97a50d6744c62a0e3f6a358e573ff31d07bc2eb700d3753c589"
Nov 02 18:00:23.072723 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:23.072708    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr_odh-ai-gateway-infra(5e580309-6fbb-48e4-9f1e-93cba517df80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80"
Nov 02 18:00:23.082449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.082427    4956 scope.go:122] "RemoveContainer" containerID="f507d6dc91e8ce0b28cf8878ed7f4739dc641fb3123b8a2d939e83d07f315c52"
Nov 02 18:00:23.187198 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.187162    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"]
Nov 02 18:00:23.350615 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.350591    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:23.449916 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.449880    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725c3153-c34e-49e8-ada1-af6d759c6d1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c3153-c34e-49e8-ada1-af6d759c6d1e-maas-api-service-ca\") pod \"725c3153-c34e-49e8-ada1-af6d759c6d1e\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") "
Nov 02 18:00:23.450109 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.449974    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725c3153-c34e-49e8-ada1-af6d759c6d1e-kube-api-access-2knrc\" (UniqueName: \"kubernetes.io/projected/725c3153-c34e-49e8-ada1-af6d759c6d1e-kube-api-access-2knrc\") pod \"725c3153-c34e-49e8-ada1-af6d759c6d1e\" (UID: \"725c3153-c34e-49e8-ada1-af6d759c6d1e\") "
Nov 02 18:00:23.450295 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.450269    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725c3153-c34e-49e8-ada1-af6d759c6d1e-maas-api-service-ca" pod "725c3153-c34e-49e8-ada1-af6d759c6d1e" (UID: "725c3153-c34e-49e8-ada1-af6d759c6d1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:23.450429 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.450415    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c3153-c34e-49e8-ada1-af6d759c6d1e-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:23.452268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.452244    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725c3153-c34e-49e8-ada1-af6d759c6d1e-kube-api-access-2knrc" pod "725c3153-c34e-49e8-ada1-af6d759c6d1e" (UID: "725c3153-c34e-49e8-ada1-af6d759c6d1e"). InnerVolumeSpecName "kube-api-access-2knrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:23.552011 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.551982    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knrc\" (UniqueName: \"kubernetes.io/projected/725c3153-c34e-49e8-ada1-af6d759c6d1e-kube-api-access-2knrc\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:23.857865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.857828    4956 scope.go:122] "RemoveContainer" containerID="c447436d69c82028e3c4cf18a83b38869fa565ff251b171f0ec365a82ddb98e6"
Nov 02 18:00:23.858044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:23.857924    4956 scope.go:122] "RemoveContainer" containerID="cb34cd80b23dc10371070e2d2137dd68641f6d117edd6fc9b6f50b8ef0267e46"
Nov 02 18:00:24.082868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.082831    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw" event={"ID":"725c3153-c34e-49e8-ada1-af6d759c6d1e","Type":"ContainerDied","Data":"085ed0f8ede10efe082f5b7b201323a796aea501f57e3a5eba3477600cbd65bd"}
Nov 02 18:00:24.083359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.082873    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"
Nov 02 18:00:24.083359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.082898    4956 scope.go:122] "RemoveContainer" containerID="60be642017ce432586ee18c38a227a08131f4199fba871bc73eeee5022d80874"
Nov 02 18:00:24.199738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.199706    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"]
Nov 02 18:00:24.246621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.246586    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"]
Nov 02 18:00:24.254086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.254053    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894040-vjzjw"]
Nov 02 18:00:24.323503 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.323480    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:24.361740 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.361713    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ef0ef6-7486-400d-a613-7f5aaa389aae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef0ef6-7486-400d-a613-7f5aaa389aae-maas-api-service-ca\") pod \"07ef0ef6-7486-400d-a613-7f5aaa389aae\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") "
Nov 02 18:00:24.361901 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.361817    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ef0ef6-7486-400d-a613-7f5aaa389aae-kube-api-access-2hqxp\" (UniqueName: \"kubernetes.io/projected/07ef0ef6-7486-400d-a613-7f5aaa389aae-kube-api-access-2hqxp\") pod \"07ef0ef6-7486-400d-a613-7f5aaa389aae\" (UID: \"07ef0ef6-7486-400d-a613-7f5aaa389aae\") "
Nov 02 18:00:24.362095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.362070    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ef0ef6-7486-400d-a613-7f5aaa389aae-maas-api-service-ca" pod "07ef0ef6-7486-400d-a613-7f5aaa389aae" (UID: "07ef0ef6-7486-400d-a613-7f5aaa389aae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:24.425826 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.425663    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ef0ef6-7486-400d-a613-7f5aaa389aae-kube-api-access-2hqxp" pod "07ef0ef6-7486-400d-a613-7f5aaa389aae" (UID: "07ef0ef6-7486-400d-a613-7f5aaa389aae"). InnerVolumeSpecName "kube-api-access-2hqxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:24.429458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.429432    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"]
Nov 02 18:00:24.463101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.463060    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqxp\" (UniqueName: \"kubernetes.io/projected/07ef0ef6-7486-400d-a613-7f5aaa389aae-kube-api-access-2hqxp\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:24.463101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.463098    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ef0ef6-7486-400d-a613-7f5aaa389aae-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:24.602541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.602507    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:24.665758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.665723    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e580309-6fbb-48e4-9f1e-93cba517df80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e580309-6fbb-48e4-9f1e-93cba517df80-maas-api-service-ca\") pod \"5e580309-6fbb-48e4-9f1e-93cba517df80\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") "
Nov 02 18:00:24.665916 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.665871    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e580309-6fbb-48e4-9f1e-93cba517df80-kube-api-access-cd2n4\" (UniqueName: \"kubernetes.io/projected/5e580309-6fbb-48e4-9f1e-93cba517df80-kube-api-access-cd2n4\") pod \"5e580309-6fbb-48e4-9f1e-93cba517df80\" (UID: \"5e580309-6fbb-48e4-9f1e-93cba517df80\") "
Nov 02 18:00:24.666203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.666180    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e580309-6fbb-48e4-9f1e-93cba517df80-maas-api-service-ca" pod "5e580309-6fbb-48e4-9f1e-93cba517df80" (UID: "5e580309-6fbb-48e4-9f1e-93cba517df80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:24.666326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.666314    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e580309-6fbb-48e4-9f1e-93cba517df80-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:24.668210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.668188    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e580309-6fbb-48e4-9f1e-93cba517df80-kube-api-access-cd2n4" pod "5e580309-6fbb-48e4-9f1e-93cba517df80" (UID: "5e580309-6fbb-48e4-9f1e-93cba517df80"). InnerVolumeSpecName "kube-api-access-cd2n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:24.767004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:24.766963    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd2n4\" (UniqueName: \"kubernetes.io/projected/5e580309-6fbb-48e4-9f1e-93cba517df80-kube-api-access-cd2n4\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:25.090813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.090772    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"
Nov 02 18:00:25.090813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.090799    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4" event={"ID":"07ef0ef6-7486-400d-a613-7f5aaa389aae","Type":"ContainerDied","Data":"d24af16876dbf139ace7ad009437bf32b8da37f247f8ad1f50d8e8f1cb8d3075"}
Nov 02 18:00:25.091401 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.090864    4956 scope.go:122] "RemoveContainer" containerID="c7ac81f80102373fb62428c8a5b851b810889e2aca3d393c21ee879d2f69d812"
Nov 02 18:00:25.092547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.092509    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr" event={"ID":"5e580309-6fbb-48e4-9f1e-93cba517df80","Type":"ContainerDied","Data":"b9affb1f2356bfd81fe8e08b94efa84d8b8fb1774c26308cfba4e64e6e6eead1"}
Nov 02 18:00:25.092657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.092638    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"
Nov 02 18:00:25.094646 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.094625    4956 generic.go:356] "Generic (PLEG): container finished" podID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerID="7d675c74259ff247924b55909ed75fbbd4621995a14f70bdd52ddf442061a8f2" exitCode=60
Nov 02 18:00:25.094763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.094648    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" event={"ID":"41479ffc-a4e0-40f4-a6db-72efe1e31c10","Type":"ContainerDied","Data":"7d675c74259ff247924b55909ed75fbbd4621995a14f70bdd52ddf442061a8f2"}
Nov 02 18:00:25.095057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.095039    4956 scope.go:122] "RemoveContainer" containerID="7d675c74259ff247924b55909ed75fbbd4621995a14f70bdd52ddf442061a8f2"
Nov 02 18:00:25.095372 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:25.095347    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-g9qqx_odh-ai-gateway-infra(41479ffc-a4e0-40f4-a6db-72efe1e31c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10"
Nov 02 18:00:25.096997 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.096978    4956 generic.go:356] "Generic (PLEG): container finished" podID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerID="5155ecf25a1696ad57b32099e42fd6582023dfe78ba30450be9ee0dd17dba374" exitCode=60
Nov 02 18:00:25.097082 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.097044    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" event={"ID":"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac","Type":"ContainerDied","Data":"5155ecf25a1696ad57b32099e42fd6582023dfe78ba30450be9ee0dd17dba374"}
Nov 02 18:00:25.097432 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.097415    4956 scope.go:122] "RemoveContainer" containerID="5155ecf25a1696ad57b32099e42fd6582023dfe78ba30450be9ee0dd17dba374"
Nov 02 18:00:25.097691 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:25.097673    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm_odh-ai-gateway-infra(4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac"
Nov 02 18:00:25.102736 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.102621    4956 scope.go:122] "RemoveContainer" containerID="d05025dbae92f97a50d6744c62a0e3f6a358e573ff31d07bc2eb700d3753c589"
Nov 02 18:00:25.111588 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.111567    4956 scope.go:122] "RemoveContainer" containerID="c447436d69c82028e3c4cf18a83b38869fa565ff251b171f0ec365a82ddb98e6"
Nov 02 18:00:25.124160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.124135    4956 scope.go:122] "RemoveContainer" containerID="cb34cd80b23dc10371070e2d2137dd68641f6d117edd6fc9b6f50b8ef0267e46"
Nov 02 18:00:25.369924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.369827    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"]
Nov 02 18:00:25.375807 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.375780    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894040-pf9nr"]
Nov 02 18:00:25.486224 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.486192    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"]
Nov 02 18:00:25.490363 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.490335    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-943ca1-29894040-r92x4"]
Nov 02 18:00:25.864901 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.864868    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" path="/var/lib/kubelet/pods/07ef0ef6-7486-400d-a613-7f5aaa389aae/volumes"
Nov 02 18:00:25.865328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.865314    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" path="/var/lib/kubelet/pods/5e580309-6fbb-48e4-9f1e-93cba517df80/volumes"
Nov 02 18:00:25.865683 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:25.865670    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" path="/var/lib/kubelet/pods/725c3153-c34e-49e8-ada1-af6d759c6d1e/volumes"
Nov 02 18:00:26.269671 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.269637    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"]
Nov 02 18:00:26.427359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.427331    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:26.487725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.487690    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-maas-api-service-ca\") pod \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") "
Nov 02 18:00:26.487927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.487795    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-kube-api-access-6f8vb\" (UniqueName: \"kubernetes.io/projected/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-kube-api-access-6f8vb\") pod \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\" (UID: \"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac\") "
Nov 02 18:00:26.488061 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.488037    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-maas-api-service-ca" pod "4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" (UID: "4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:26.488170 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.488156    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:26.490062 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.490037    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-kube-api-access-6f8vb" pod "4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" (UID: "4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac"). InnerVolumeSpecName "kube-api-access-6f8vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:26.589509 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.589426    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f8vb\" (UniqueName: \"kubernetes.io/projected/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac-kube-api-access-6f8vb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:26.618090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.618052    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"]
Nov 02 18:00:26.822914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.822890    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:26.899728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.899643    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41479ffc-a4e0-40f4-a6db-72efe1e31c10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41479ffc-a4e0-40f4-a6db-72efe1e31c10-maas-api-service-ca\") pod \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") "
Nov 02 18:00:26.899728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.899680    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41479ffc-a4e0-40f4-a6db-72efe1e31c10-kube-api-access-n2g45\" (UniqueName: \"kubernetes.io/projected/41479ffc-a4e0-40f4-a6db-72efe1e31c10-kube-api-access-n2g45\") pod \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\" (UID: \"41479ffc-a4e0-40f4-a6db-72efe1e31c10\") "
Nov 02 18:00:26.900013 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.899988    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41479ffc-a4e0-40f4-a6db-72efe1e31c10-maas-api-service-ca" pod "41479ffc-a4e0-40f4-a6db-72efe1e31c10" (UID: "41479ffc-a4e0-40f4-a6db-72efe1e31c10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:26.900097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.900084    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41479ffc-a4e0-40f4-a6db-72efe1e31c10-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:26.901979 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:26.901957    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41479ffc-a4e0-40f4-a6db-72efe1e31c10-kube-api-access-n2g45" pod "41479ffc-a4e0-40f4-a6db-72efe1e31c10" (UID: "41479ffc-a4e0-40f4-a6db-72efe1e31c10"). InnerVolumeSpecName "kube-api-access-n2g45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:27.001086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.001051    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2g45\" (UniqueName: \"kubernetes.io/projected/41479ffc-a4e0-40f4-a6db-72efe1e31c10-kube-api-access-n2g45\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:27.112196 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112162    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"]
Nov 02 18:00:27.112671 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112658    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.112712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112672    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.112712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112684    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112689    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112700    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112707    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112714    4956 cpu_manager.go:414] "containerMap: removing container" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112719    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112728    4956 cpu_manager.go:414] "containerMap: removing container" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112733    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112742    4956 cpu_manager.go:414] "containerMap: removing container" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112747    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112753    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112757    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112769    4956 cpu_manager.go:414] "containerMap: removing container" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112774    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112787    4956 cpu_manager.go:414] "containerMap: removing container" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112791    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112797    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112802    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112818    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112823    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112833    4956 cpu_manager.go:414] "containerMap: removing container" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112838    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112914    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112924    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112930    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112936    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:00:27.112933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112942    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112948    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112954    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112960    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112965    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112970    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112977    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112982    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:00:27.113597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.112989    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e580309-6fbb-48e4-9f1e-93cba517df80" containerName="cleanup"
Nov 02 18:00:27.116220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.116193    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"
Nov 02 18:00:27.116586 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.116549    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx" event={"ID":"41479ffc-a4e0-40f4-a6db-72efe1e31c10","Type":"ContainerDied","Data":"447d5b6b0aa9961c73e5a5f77ab042521cad05a3665bb3b658170cd7cc6befc1"}
Nov 02 18:00:27.116681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.116600    4956 scope.go:122] "RemoveContainer" containerID="7d675c74259ff247924b55909ed75fbbd4621995a14f70bdd52ddf442061a8f2"
Nov 02 18:00:27.116681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.116670    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.117836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.117812    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm" event={"ID":"4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac","Type":"ContainerDied","Data":"cb3c7ac5804bae296648944937e8175471cd4aa91944b2db56c6093d94496e95"}
Nov 02 18:00:27.117949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.117821    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"
Nov 02 18:00:27.127857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.127839    4956 scope.go:122] "RemoveContainer" containerID="5155ecf25a1696ad57b32099e42fd6582023dfe78ba30450be9ee0dd17dba374"
Nov 02 18:00:27.171799 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.171689    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f07b5b-serving-cert\""
Nov 02 18:00:27.175142 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.175100    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"]
Nov 02 18:00:27.203107 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.203070    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgp7\" (UniqueName: \"kubernetes.io/projected/698f8516-39a5-464f-93ed-7f2aa1bbdaee-kube-api-access-fzgp7\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.203286 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.203113    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698f8516-39a5-464f-93ed-7f2aa1bbdaee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.203286 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.203230    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-metrics-certs\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.203434 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.203411    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.304249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.304219    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgp7\" (UniqueName: \"kubernetes.io/projected/698f8516-39a5-464f-93ed-7f2aa1bbdaee-kube-api-access-fzgp7\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.304249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.304254    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698f8516-39a5-464f-93ed-7f2aa1bbdaee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.304742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.304279    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-metrics-certs\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.304742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.304409    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.304742 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:27.304546    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-serving-cert: secret "maas-api-e2e-ipp-a-f07b5b-serving-cert" not found
Nov 02 18:00:27.304742 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:27.304628    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls podName:698f8516-39a5-464f-93ed-7f2aa1bbdaee nodeName:}" failed. No retries permitted until 2026-11-02 18:00:27.804607463 +0000 UTC m=+2258.061638079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls") pod "maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr" (UID: "698f8516-39a5-464f-93ed-7f2aa1bbdaee") : secret "maas-api-e2e-ipp-a-f07b5b-serving-cert" not found
Nov 02 18:00:27.304977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.304960    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698f8516-39a5-464f-93ed-7f2aa1bbdaee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.306919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.306891    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-metrics-certs\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.365107 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.365058    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgp7\" (UniqueName: \"kubernetes.io/projected/698f8516-39a5-464f-93ed-7f2aa1bbdaee-kube-api-access-fzgp7\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.395068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.395033    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"]
Nov 02 18:00:27.399483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.399453    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-g9qqx"]
Nov 02 18:00:27.508284 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.508246    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"]
Nov 02 18:00:27.513342 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.513311    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5924c-29894040-w2pmm"]
Nov 02 18:00:27.811387 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.811304    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.814059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.814029    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:27.866824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.866782    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" path="/var/lib/kubelet/pods/41479ffc-a4e0-40f4-a6db-72efe1e31c10/volumes"
Nov 02 18:00:27.867311 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:27.867291    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac" path="/var/lib/kubelet/pods/4e14e8cc-e7a0-4701-8ac2-7c3ff57e95ac/volumes"
Nov 02 18:00:28.035945 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:28.035909    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:28.424727 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:28.424691    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"]
Nov 02 18:00:28.426486 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:00:28.426445    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698f8516_39a5_464f_93ed_7f2aa1bbdaee.slice/crio-2db4a08d3c465e14ecc0ef6861b0d161c9e49223ff486ccacf69eb747eab2ead WatchSource:0}: Error finding container 2db4a08d3c465e14ecc0ef6861b0d161c9e49223ff486ccacf69eb747eab2ead: Status 404 returned error can't find the container with id 2db4a08d3c465e14ecc0ef6861b0d161c9e49223ff486ccacf69eb747eab2ead
Nov 02 18:00:29.055336 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:29.055296    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:00:29.132869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:29.132827    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr" event={"ID":"698f8516-39a5-464f-93ed-7f2aa1bbdaee","Type":"ContainerStarted","Data":"2db4a08d3c465e14ecc0ef6861b0d161c9e49223ff486ccacf69eb747eab2ead"}
Nov 02 18:00:30.141509 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:30.141469    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr" event={"ID":"698f8516-39a5-464f-93ed-7f2aa1bbdaee","Type":"ContainerStarted","Data":"249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7"}
Nov 02 18:00:30.142019 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:30.141540    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:30.260220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:30.260165    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr" podStartSLOduration=2.199242522 podStartE2EDuration="3.260150767s" totalImagesPullingTime="1.060908245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:30.256690054 +0000 UTC m=+2260.513720674" watchObservedRunningTime="2026-11-02 18:00:30.260150767 +0000 UTC m=+2260.517181378"
Nov 02 18:00:36.153044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:00:36.153016    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:00:59.053016 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:00:59.052979    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:01:00.923045 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.922959    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"]
Nov 02 18:01:00.923748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923726    4956 cpu_manager.go:414] "containerMap: removing container" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:01:00.923842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923750    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:01:00.923842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923769    4956 cpu_manager.go:414] "containerMap: removing container" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:01:00.923842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923776    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41479ffc-a4e0-40f4-a6db-72efe1e31c10" containerName="cleanup"
Nov 02 18:01:00.923842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923801    4956 cpu_manager.go:414] "containerMap: removing container" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:01:00.923842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923809    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:01:00.924088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923975    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="725c3153-c34e-49e8-ada1-af6d759c6d1e" containerName="cleanup"
Nov 02 18:01:00.924088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.923991    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ef0ef6-7486-400d-a613-7f5aaa389aae" containerName="cleanup"
Nov 02 18:01:00.928968 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.928945    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.981542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.981511    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cf2b8a-openshift-default-dockercfg-x9cbg\""
Nov 02 18:01:00.988503 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.988449    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92f5015-8f4d-4c26-94d0-b30257b5183b-istiod-ca-cert\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.988866 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.988843    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-podinfo\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.989089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.989072    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-token\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.989286 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.989260    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-certs\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.989406 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.989351    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lcs\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-kube-api-access-86lcs\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.989486 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.989471    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-socket\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.989546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.989507    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-credential-socket\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.989604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.989547    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-data\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:00.989604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:00.989584    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-envoy\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.033458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.033423    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"]
Nov 02 18:01:01.090383 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090337    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-socket\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090405    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-credential-socket\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090449    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-data\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090486    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-envoy\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090548    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92f5015-8f4d-4c26-94d0-b30257b5183b-istiod-ca-cert\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090591    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-podinfo\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090653    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-token\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090703    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-certs\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090830    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-socket\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.090994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.090987    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-credential-socket\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.091259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.091221    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lcs\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-kube-api-access-86lcs\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.091346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.091250    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92f5015-8f4d-4c26-94d0-b30257b5183b-istiod-ca-cert\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.091411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.091355    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-data\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.091411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.091402    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-certs\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.093492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.093467    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-envoy\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.093492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.093475    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-podinfo\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.148384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.148348    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lcs\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-kube-api-access-86lcs\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.148595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.148568    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-token\") pod \"e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") " pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.242551 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.242519    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:01.680078 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.680042    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"]
Nov 02 18:01:01.681647 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:01:01.681619    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92f5015_8f4d_4c26_94d0_b30257b5183b.slice/crio-d6f4dc72a1c6135d33fecc51c34948c3058bd5c978e26568218be21c81b9e8fc WatchSource:0}: Error finding container d6f4dc72a1c6135d33fecc51c34948c3058bd5c978e26568218be21c81b9e8fc: Status 404 returned error can't find the container with id d6f4dc72a1c6135d33fecc51c34948c3058bd5c978e26568218be21c81b9e8fc
Nov 02 18:01:01.684228 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.684192    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:01.684332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.684276    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:01.684332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:01.684322    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:02.343688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:02.343647    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" event={"ID":"e92f5015-8f4d-4c26-94d0-b30257b5183b","Type":"ContainerStarted","Data":"77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d"}
Nov 02 18:01:02.344189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:02.343708    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" event={"ID":"e92f5015-8f4d-4c26-94d0-b30257b5183b","Type":"ContainerStarted","Data":"d6f4dc72a1c6135d33fecc51c34948c3058bd5c978e26568218be21c81b9e8fc"}
Nov 02 18:01:02.460667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:02.460616    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" podStartSLOduration=2.460599718 podStartE2EDuration="2.460599718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:02.458354201 +0000 UTC m=+2292.715384820" watchObservedRunningTime="2026-11-02 18:01:02.460599718 +0000 UTC m=+2292.717630338"
Nov 02 18:01:03.243318 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:03.243276    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:03.249197 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:03.249168    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:03.349018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:03.348989    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:03.349993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:03.349977    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:01:09.164153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.161316    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"]
Nov 02 18:01:09.167155 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.166870    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.220216 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.220169    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"]
Nov 02 18:01:09.266056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.266030    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cf2b8a\""
Nov 02 18:01:09.266271 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.266109    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cf2b8a-dockercfg-q5hxt\""
Nov 02 18:01:09.304504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.304468    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.304668 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.304614    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzsx\" (UniqueName: \"kubernetes.io/projected/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-kube-api-access-7zzsx\") pod \"payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.405967 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.405931    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.406201 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.406045    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzsx\" (UniqueName: \"kubernetes.io/projected/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-kube-api-access-7zzsx\") pod \"payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.406717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.406688    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.463664 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.463630    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzsx\" (UniqueName: \"kubernetes.io/projected/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-kube-api-access-7zzsx\") pod \"payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.484094 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.484058    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:09.865269 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:01:09.865243    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a17c2ff_3427_4c72_bfcf_d4f2ea9bf934.slice/crio-a839fe37e8398bb623024a2f4d28cf8c4ed337240eb75b12e537b77fdb0d795b WatchSource:0}: Error finding container a839fe37e8398bb623024a2f4d28cf8c4ed337240eb75b12e537b77fdb0d795b: Status 404 returned error can't find the container with id a839fe37e8398bb623024a2f4d28cf8c4ed337240eb75b12e537b77fdb0d795b
Nov 02 18:01:09.867655 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:09.867633    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"]
Nov 02 18:01:10.392584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:10.392552    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n" event={"ID":"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934","Type":"ContainerStarted","Data":"656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5"}
Nov 02 18:01:10.393006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:10.392604    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n" event={"ID":"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934","Type":"ContainerStarted","Data":"a839fe37e8398bb623024a2f4d28cf8c4ed337240eb75b12e537b77fdb0d795b"}
Nov 02 18:01:10.393006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:10.392656    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:10.507061 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:10.506994    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n" podStartSLOduration=1.506976994 podStartE2EDuration="1.506976994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:10.506285604 +0000 UTC m=+2300.763316223" watchObservedRunningTime="2026-11-02 18:01:10.506976994 +0000 UTC m=+2300.764007614"
Nov 02 18:01:21.401634 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:21.401598    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:01:23.330577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:23.330543    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"]
Nov 02 18:01:28.331112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.331063    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b" podUID="24e6cb24-e10d-423d-8740-a7ebbc0a1138" containerName="payload-processing" containerID="cri-o://b8ec7d619083000f24ee2f8154a17913bec32018f2681f74b531a94a8f22df1f" gracePeriod=25
Nov 02 18:01:28.507196 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.507157    4956 generic.go:356] "Generic (PLEG): container finished" podID="24e6cb24-e10d-423d-8740-a7ebbc0a1138" containerID="b8ec7d619083000f24ee2f8154a17913bec32018f2681f74b531a94a8f22df1f" exitCode=0
Nov 02 18:01:28.507377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.507237    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b" event={"ID":"24e6cb24-e10d-423d-8740-a7ebbc0a1138","Type":"ContainerDied","Data":"b8ec7d619083000f24ee2f8154a17913bec32018f2681f74b531a94a8f22df1f"}
Nov 02 18:01:28.606181 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.606150    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 18:01:28.733321 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.733285    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e6cb24-e10d-423d-8740-a7ebbc0a1138-kube-api-access-5fhnm\" (UniqueName: \"kubernetes.io/projected/24e6cb24-e10d-423d-8740-a7ebbc0a1138-kube-api-access-5fhnm\") pod \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") "
Nov 02 18:01:28.733511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.733498    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e6cb24-e10d-423d-8740-a7ebbc0a1138-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e6cb24-e10d-423d-8740-a7ebbc0a1138-plugins-config-volume\") pod \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\" (UID: \"24e6cb24-e10d-423d-8740-a7ebbc0a1138\") "
Nov 02 18:01:28.733933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.733900    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e6cb24-e10d-423d-8740-a7ebbc0a1138-plugins-config-volume" pod "24e6cb24-e10d-423d-8740-a7ebbc0a1138" (UID: "24e6cb24-e10d-423d-8740-a7ebbc0a1138"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:28.736239 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.736209    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e6cb24-e10d-423d-8740-a7ebbc0a1138-kube-api-access-5fhnm" pod "24e6cb24-e10d-423d-8740-a7ebbc0a1138" (UID: "24e6cb24-e10d-423d-8740-a7ebbc0a1138"). InnerVolumeSpecName "kube-api-access-5fhnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:28.835504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.835468    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e6cb24-e10d-423d-8740-a7ebbc0a1138-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:28.835504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:28.835498    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fhnm\" (UniqueName: \"kubernetes.io/projected/24e6cb24-e10d-423d-8740-a7ebbc0a1138-kube-api-access-5fhnm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:29.052274 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:01:29.052236    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:01:29.513764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:29.513730    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b" event={"ID":"24e6cb24-e10d-423d-8740-a7ebbc0a1138","Type":"ContainerDied","Data":"42663da80f27550d971166cf845e82bb771cccb5c585cda77cac4313ecee7cc7"}
Nov 02 18:01:29.513764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:29.513748    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"
Nov 02 18:01:29.514358 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:29.513796    4956 scope.go:122] "RemoveContainer" containerID="b8ec7d619083000f24ee2f8154a17913bec32018f2681f74b531a94a8f22df1f"
Nov 02 18:01:29.676801 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:29.676762    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"]
Nov 02 18:01:29.727086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:29.727053    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5a1e27-64ff4cf7c-hcl4b"]
Nov 02 18:01:29.864754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:29.864678    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e6cb24-e10d-423d-8740-a7ebbc0a1138" path="/var/lib/kubelet/pods/24e6cb24-e10d-423d-8740-a7ebbc0a1138/volumes"
Nov 02 18:01:39.283476 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.283443    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"]
Nov 02 18:01:39.284053 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.283631    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr" podUID="0e46110b-6557-491a-84d6-eca268d92d51" containerName="discovery" containerID="cri-o://be39a54c13b4b6c23bc02fa6431beff5c384b12f221732d39ec9ee4912a32c56" gracePeriod=30
Nov 02 18:01:39.578864 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.578827    4956 generic.go:356] "Generic (PLEG): container finished" podID="0e46110b-6557-491a-84d6-eca268d92d51" containerID="be39a54c13b4b6c23bc02fa6431beff5c384b12f221732d39ec9ee4912a32c56" exitCode=0
Nov 02 18:01:39.579029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.578902    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr" event={"ID":"0e46110b-6557-491a-84d6-eca268d92d51","Type":"ContainerDied","Data":"be39a54c13b4b6c23bc02fa6431beff5c384b12f221732d39ec9ee4912a32c56"}
Nov 02 18:01:39.579029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.578963    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr" event={"ID":"0e46110b-6557-491a-84d6-eca268d92d51","Type":"ContainerDied","Data":"edcfbe2271290c8c64c295dd77520031030c322289367c642e3e23b421e96c9f"}
Nov 02 18:01:39.579029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.578984    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edcfbe2271290c8c64c295dd77520031030c322289367c642e3e23b421e96c9f"
Nov 02 18:01:39.580906 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.580886    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 18:01:39.663256 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.663222    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-dns-cert\") pod \"0e46110b-6557-491a-84d6-eca268d92d51\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") "
Nov 02 18:01:39.663438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.663280    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-kubeconfig\") pod \"0e46110b-6557-491a-84d6-eca268d92d51\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") "
Nov 02 18:01:39.663438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.663357    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-istio-token\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-istio-token\") pod \"0e46110b-6557-491a-84d6-eca268d92d51\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") "
Nov 02 18:01:39.663438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.663391    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-cacerts\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-cacerts\") pod \"0e46110b-6557-491a-84d6-eca268d92d51\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") "
Nov 02 18:01:39.663438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.663417    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e46110b-6557-491a-84d6-eca268d92d51-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46110b-6557-491a-84d6-eca268d92d51-local-certs\") pod \"0e46110b-6557-491a-84d6-eca268d92d51\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") "
Nov 02 18:01:39.663595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.663492    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-kube-api-access-q4r9t\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-kube-api-access-q4r9t\") pod \"0e46110b-6557-491a-84d6-eca268d92d51\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") "
Nov 02 18:01:39.663595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.663533    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-ca-configmap\") pod \"0e46110b-6557-491a-84d6-eca268d92d51\" (UID: \"0e46110b-6557-491a-84d6-eca268d92d51\") "
Nov 02 18:01:39.664267 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.664204    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-ca-configmap" pod "0e46110b-6557-491a-84d6-eca268d92d51" (UID: "0e46110b-6557-491a-84d6-eca268d92d51"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:39.665946 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.665912    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-kube-api-access-q4r9t" pod "0e46110b-6557-491a-84d6-eca268d92d51" (UID: "0e46110b-6557-491a-84d6-eca268d92d51"). InnerVolumeSpecName "kube-api-access-q4r9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:39.666036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.665966    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e46110b-6557-491a-84d6-eca268d92d51-local-certs" pod "0e46110b-6557-491a-84d6-eca268d92d51" (UID: "0e46110b-6557-491a-84d6-eca268d92d51"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:39.666300 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.666280    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-dns-cert" pod "0e46110b-6557-491a-84d6-eca268d92d51" (UID: "0e46110b-6557-491a-84d6-eca268d92d51"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:39.666409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.666329    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-istio-token" pod "0e46110b-6557-491a-84d6-eca268d92d51" (UID: "0e46110b-6557-491a-84d6-eca268d92d51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:39.666409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.666389    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-kubeconfig" pod "0e46110b-6557-491a-84d6-eca268d92d51" (UID: "0e46110b-6557-491a-84d6-eca268d92d51"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:39.666496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.666462    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-cacerts" pod "0e46110b-6557-491a-84d6-eca268d92d51" (UID: "0e46110b-6557-491a-84d6-eca268d92d51"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:39.764734 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.764696    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4r9t\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-kube-api-access-q4r9t\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:39.764734 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.764729    4956 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-ca-configmap\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:39.764734 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.764740    4956 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-csr-dns-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:39.764985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.764751    4956 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-istio-kubeconfig\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:39.764985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.764760    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46110b-6557-491a-84d6-eca268d92d51-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:39.764985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.764770    4956 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e46110b-6557-491a-84d6-eca268d92d51-cacerts\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:39.764985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:39.764779    4956 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46110b-6557-491a-84d6-eca268d92d51-local-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:01:40.585159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:40.585108    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"
Nov 02 18:01:40.799575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:40.799539    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"]
Nov 02 18:01:40.848977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:40.848894    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9hcr"]
Nov 02 18:01:41.865931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:41.865891    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e46110b-6557-491a-84d6-eca268d92d51" path="/var/lib/kubelet/pods/0e46110b-6557-491a-84d6-eca268d92d51/volumes"
Nov 02 18:01:50.422692 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:50.422658    4956 scope.go:122] "RemoveContainer" containerID="be39a54c13b4b6c23bc02fa6431beff5c384b12f221732d39ec9ee4912a32c56"
Nov 02 18:01:51.617863 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:51.617832    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gtcjl_8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44/payload-processing/0.log"
Nov 02 18:01:51.874172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:01:51.874064    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gtcjl_8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44/payload-processing/0.log"
Nov 02 18:01:59.053716 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:01:59.053677    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:02:01.281263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:01.281223    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"]
Nov 02 18:02:01.281737 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:01.281467    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" containerID="cri-o://681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896" gracePeriod=30
Nov 02 18:02:02.396516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:02.396478    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-drckw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:02:02.396924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:02.396564    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:02:06.551095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.551019    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 18:02:06.682085 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682053    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-data\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682151    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-certs\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682203    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-kube-api-access-khfxd\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-kube-api-access-khfxd\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682228    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-token\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-token\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682274    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-credential-socket\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682435    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-data" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:06.682562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682456    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-certs" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:06.682562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682476    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-socket\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682511    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-envoy\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682600    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-podinfo\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682625    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-credential-socket" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:06.682780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682640    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bd9608-761b-4a88-8641-4a56fb5d092c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bd9608-761b-4a88-8641-4a56fb5d092c-istiod-ca-cert\") pod \"98bd9608-761b-4a88-8641-4a56fb5d092c\" (UID: \"98bd9608-761b-4a88-8641-4a56fb5d092c\") "
Nov 02 18:02:06.682780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.682743    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-socket" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:06.683081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.683032    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.683081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.683057    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.683081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.683026    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bd9608-761b-4a88-8641-4a56fb5d092c-istiod-ca-cert" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:06.683081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.683067    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.683528 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.683109    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.684795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.684771    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-envoy" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:06.684917 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.684859    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-kube-api-access-khfxd" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "kube-api-access-khfxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:06.684917 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.684891    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-token" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:06.684996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.684950    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-podinfo" pod "98bd9608-761b-4a88-8641-4a56fb5d092c" (UID: "98bd9608-761b-4a88-8641-4a56fb5d092c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:02:06.755404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.755367    4956 generic.go:356] "Generic (PLEG): container finished" podID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerID="681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896" exitCode=0
Nov 02 18:02:06.755608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.755414    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" event={"ID":"98bd9608-761b-4a88-8641-4a56fb5d092c","Type":"ContainerDied","Data":"681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896"}
Nov 02 18:02:06.755608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.755431    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"
Nov 02 18:02:06.755608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.755510    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw" event={"ID":"98bd9608-761b-4a88-8641-4a56fb5d092c","Type":"ContainerDied","Data":"ba262dcb88208d615b0af1fddd1dc3b682393faed4e18d05dd37830a25ab5de2"}
Nov 02 18:02:06.755608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.755542    4956 scope.go:122] "RemoveContainer" containerID="681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896"
Nov 02 18:02:06.770281 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.770257    4956 scope.go:122] "RemoveContainer" containerID="681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896"
Nov 02 18:02:06.770598 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:02:06.770577    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896\": container with ID starting with 681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896 not found: ID does not exist" containerID="681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896"
Nov 02 18:02:06.770660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.770608    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896"} err="failed to get container status \"681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896\": rpc error: code = NotFound desc = could not find container \"681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896\": container with ID starting with 681ac5c86cabcbed3676be303e329143321de053ab4231c1e6525c06c8543896 not found: ID does not exist"
Nov 02 18:02:06.784722 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.784695    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khfxd\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-kube-api-access-khfxd\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.784722 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.784725    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.784899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.784737    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.784899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.784746    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bd9608-761b-4a88-8641-4a56fb5d092c-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.784899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.784755    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bd9608-761b-4a88-8641-4a56fb5d092c-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:06.880416 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.880333    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"]
Nov 02 18:02:06.883771 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:06.883743    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-drckw"]
Nov 02 18:02:07.864667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:07.864634    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" path="/var/lib/kubelet/pods/98bd9608-761b-4a88-8641-4a56fb5d092c/volumes"
Nov 02 18:02:12.994648 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.994566    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"]
Nov 02 18:02:12.995493 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995473    4956 cpu_manager.go:414] "containerMap: removing container" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy"
Nov 02 18:02:12.995546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995498    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy"
Nov 02 18:02:12.995546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995527    4956 cpu_manager.go:414] "containerMap: removing container" podUID="0e46110b-6557-491a-84d6-eca268d92d51" containerName="discovery"
Nov 02 18:02:12.995546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995535    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e46110b-6557-491a-84d6-eca268d92d51" containerName="discovery"
Nov 02 18:02:12.995641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995554    4956 cpu_manager.go:414] "containerMap: removing container" podUID="24e6cb24-e10d-423d-8740-a7ebbc0a1138" containerName="payload-processing"
Nov 02 18:02:12.995641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995562    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e6cb24-e10d-423d-8740-a7ebbc0a1138" containerName="payload-processing"
Nov 02 18:02:12.995709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995696    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bd9608-761b-4a88-8641-4a56fb5d092c" containerName="istio-proxy"
Nov 02 18:02:12.995745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995717    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e46110b-6557-491a-84d6-eca268d92d51" containerName="discovery"
Nov 02 18:02:12.995745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:12.995728    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e6cb24-e10d-423d-8740-a7ebbc0a1138" containerName="payload-processing"
Nov 02 18:02:13.001224 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.001196    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.106320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.106283    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cf2b8a-serving-cert\""
Nov 02 18:02:13.108972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.108948    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"]
Nov 02 18:02:13.161345 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.161305    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.161521 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.161375    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a5089c-912e-476a-8298-e0a62e317a1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.161521 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.161496    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.161607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.161564    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982bf\" (UniqueName: \"kubernetes.io/projected/45a5089c-912e-476a-8298-e0a62e317a1b-kube-api-access-982bf\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.262743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.262629    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.262743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.262713    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982bf\" (UniqueName: \"kubernetes.io/projected/45a5089c-912e-476a-8298-e0a62e317a1b-kube-api-access-982bf\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.263411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.263380    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.263542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.263450    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a5089c-912e-476a-8298-e0a62e317a1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.264172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.264145    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a5089c-912e-476a-8298-e0a62e317a1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.265828 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.265798    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.266844 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.266823    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.322685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.322653    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-982bf\" (UniqueName: \"kubernetes.io/projected/45a5089c-912e-476a-8298-e0a62e317a1b-kube-api-access-982bf\") pod \"maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.323511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.323491    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:13.706347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.706320    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"]
Nov 02 18:02:13.707976 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:02:13.707943    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a5089c_912e_476a_8298_e0a62e317a1b.slice/crio-088ca3650704d91c92366ddcd4b3e5c1209e3d1e24960b9e38077d0ef277429d WatchSource:0}: Error finding container 088ca3650704d91c92366ddcd4b3e5c1209e3d1e24960b9e38077d0ef277429d: Status 404 returned error can't find the container with id 088ca3650704d91c92366ddcd4b3e5c1209e3d1e24960b9e38077d0ef277429d
Nov 02 18:02:13.799791 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:13.799740    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92" event={"ID":"45a5089c-912e-476a-8298-e0a62e317a1b","Type":"ContainerStarted","Data":"088ca3650704d91c92366ddcd4b3e5c1209e3d1e24960b9e38077d0ef277429d"}
Nov 02 18:02:14.807305 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:14.807261    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92" event={"ID":"45a5089c-912e-476a-8298-e0a62e317a1b","Type":"ContainerStarted","Data":"6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906"}
Nov 02 18:02:14.807738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:14.807361    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:14.929779 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:14.929678    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92" podStartSLOduration=2.048760768 podStartE2EDuration="2.929663486s" totalImagesPullingTime="880.902718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:14.92632881 +0000 UTC m=+2365.183359431" watchObservedRunningTime="2026-11-02 18:02:14.929663486 +0000 UTC m=+2365.186694106"
Nov 02 18:02:20.521105 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:20.521069    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"]
Nov 02 18:02:20.818774 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:20.818699    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:02:25.522217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.522153    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn" podUID="1776362a-6635-47cf-b50f-589d5bd5c900" containerName="payload-pre-processing" containerID="cri-o://412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23" gracePeriod=25
Nov 02 18:02:25.800849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.800824    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 18:02:25.873317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.873282    4956 generic.go:356] "Generic (PLEG): container finished" podID="1776362a-6635-47cf-b50f-589d5bd5c900" containerID="412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23" exitCode=0
Nov 02 18:02:25.873500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.873328    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"
Nov 02 18:02:25.873500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.873370    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn" event={"ID":"1776362a-6635-47cf-b50f-589d5bd5c900","Type":"ContainerDied","Data":"412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23"}
Nov 02 18:02:25.873500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.873427    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn" event={"ID":"1776362a-6635-47cf-b50f-589d5bd5c900","Type":"ContainerDied","Data":"fb8f18e359fa360c62f48715538659a8a4f817586d853a8132e3af942f319ae8"}
Nov 02 18:02:25.873500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.873452    4956 scope.go:122] "RemoveContainer" containerID="412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23"
Nov 02 18:02:25.882885 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.882867    4956 scope.go:122] "RemoveContainer" containerID="412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23"
Nov 02 18:02:25.883207 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:02:25.883182    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23\": container with ID starting with 412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23 not found: ID does not exist" containerID="412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23"
Nov 02 18:02:25.883289 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.883216    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23"} err="failed to get container status \"412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23\": rpc error: code = NotFound desc = could not find container \"412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23\": container with ID starting with 412d04b05b87752eb65fd093552d4d4ffbda8c7863a3335368cd9506e3c05b23 not found: ID does not exist"
Nov 02 18:02:25.938015 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.937981    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1776362a-6635-47cf-b50f-589d5bd5c900-kube-api-access-5xpl7\" (UniqueName: \"kubernetes.io/projected/1776362a-6635-47cf-b50f-589d5bd5c900-kube-api-access-5xpl7\") pod \"1776362a-6635-47cf-b50f-589d5bd5c900\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") "
Nov 02 18:02:25.938206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.938053    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1776362a-6635-47cf-b50f-589d5bd5c900-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1776362a-6635-47cf-b50f-589d5bd5c900-plugins-config-volume\") pod \"1776362a-6635-47cf-b50f-589d5bd5c900\" (UID: \"1776362a-6635-47cf-b50f-589d5bd5c900\") "
Nov 02 18:02:25.938486 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.938458    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1776362a-6635-47cf-b50f-589d5bd5c900-plugins-config-volume" pod "1776362a-6635-47cf-b50f-589d5bd5c900" (UID: "1776362a-6635-47cf-b50f-589d5bd5c900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:25.940499 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:25.940474    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1776362a-6635-47cf-b50f-589d5bd5c900-kube-api-access-5xpl7" pod "1776362a-6635-47cf-b50f-589d5bd5c900" (UID: "1776362a-6635-47cf-b50f-589d5bd5c900"). InnerVolumeSpecName "kube-api-access-5xpl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:26.039855 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:26.039778    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpl7\" (UniqueName: \"kubernetes.io/projected/1776362a-6635-47cf-b50f-589d5bd5c900-kube-api-access-5xpl7\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:26.039855 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:26.039806    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1776362a-6635-47cf-b50f-589d5bd5c900-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:26.295584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:26.295489    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"]
Nov 02 18:02:26.344167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:26.344133    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f42381-796c49694c-2rqtn"]
Nov 02 18:02:27.864587 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:27.864551    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1776362a-6635-47cf-b50f-589d5bd5c900" path="/var/lib/kubelet/pods/1776362a-6635-47cf-b50f-589d5bd5c900/volumes"
Nov 02 18:02:29.052884 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:02:29.052840    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:02:41.990425 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:41.990278    4956 generic.go:356] "Generic (PLEG): container finished" podID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerID="77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d" exitCode=0
Nov 02 18:02:41.991012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:41.990420    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" event={"ID":"e92f5015-8f4d-4c26-94d0-b30257b5183b","Type":"ContainerDied","Data":"77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d"}
Nov 02 18:02:41.991012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:41.990954    4956 scope.go:122] "RemoveContainer" containerID="77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d"
Nov 02 18:02:41.992626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:41.992592    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:41.992796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:41.992688    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:41.992796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:41.992752    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:42.998078 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:42.998047    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" event={"ID":"e92f5015-8f4d-4c26-94d0-b30257b5183b","Type":"ContainerStarted","Data":"158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff"}
Nov 02 18:02:43.250206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:43.248543    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:02:43.250206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:43.248667    4956 patch_prober.go:29] interesting pod/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 18:02:43.250206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:43.248709    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 18:02:44.244891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:44.244857    4956 patch_prober.go:29] interesting pod/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 18:02:44.245659 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:44.245627    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 18:02:45.256182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:45.255270    4956 patch_prober.go:29] interesting pod/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 18:02:45.256182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:45.255328    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 18:02:46.251303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:46.251202    4956 patch_prober.go:29] interesting pod/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 18:02:46.251303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:46.251284    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 18:02:47.248146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:47.245398    4956 patch_prober.go:29] interesting pod/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 18:02:47.248146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:47.245459    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 18:02:48.247153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:48.247105    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:02:48.247638 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:48.247619    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:02:48.248361 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:48.248341    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:02:50.168176 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:50.168151    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:02:50.169848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:50.169818    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:02:57.584248 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.584213    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"]
Nov 02 18:02:57.584744 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.584730    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1776362a-6635-47cf-b50f-589d5bd5c900" containerName="payload-pre-processing"
Nov 02 18:02:57.584787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.584744    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1776362a-6635-47cf-b50f-589d5bd5c900" containerName="payload-pre-processing"
Nov 02 18:02:57.584858 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.584849    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1776362a-6635-47cf-b50f-589d5bd5c900" containerName="payload-pre-processing"
Nov 02 18:02:57.589348 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.589331    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:57.691415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.691382    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f07b5b\""
Nov 02 18:02:57.691415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.691402    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f07b5b-dockercfg-pq5tf\""
Nov 02 18:02:57.694208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.694186    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"]
Nov 02 18:02:57.728743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.728696    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57141d2-0cd9-44b4-b541-813770d37e9f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:57.728954 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.728866    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqdx\" (UniqueName: \"kubernetes.io/projected/a57141d2-0cd9-44b4-b541-813770d37e9f-kube-api-access-tfqdx\") pod \"payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:57.830216 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.830157    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqdx\" (UniqueName: \"kubernetes.io/projected/a57141d2-0cd9-44b4-b541-813770d37e9f-kube-api-access-tfqdx\") pod \"payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:57.830418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.830287    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57141d2-0cd9-44b4-b541-813770d37e9f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:57.831026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.831006    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57141d2-0cd9-44b4-b541-813770d37e9f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:57.887539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.887456    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqdx\" (UniqueName: \"kubernetes.io/projected/a57141d2-0cd9-44b4-b541-813770d37e9f-kube-api-access-tfqdx\") pod \"payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:57.902303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:57.902272    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:02:58.278968 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:58.278937    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"]
Nov 02 18:02:59.053754 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:02:59.053718    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:02:59.095559 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:02:59.095524    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x" event={"ID":"a57141d2-0cd9-44b4-b541-813770d37e9f","Type":"ContainerStarted","Data":"f0fad9962b2556a6223bdfc9b104719b1f7e7cb5745659f180c6523c2bb3bb04"}
Nov 02 18:03:01.109074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:01.109034    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x" event={"ID":"a57141d2-0cd9-44b4-b541-813770d37e9f","Type":"ContainerStarted","Data":"26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd"}
Nov 02 18:03:01.109496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:01.109113    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:03:01.225916 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:01.225848    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x" podStartSLOduration=1.967152484 podStartE2EDuration="4.225816106s" totalImagesPullingTime="2.258663622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:01.224565234 +0000 UTC m=+2411.481595858" watchObservedRunningTime="2026-11-02 18:03:01.225816106 +0000 UTC m=+2411.482846727"
Nov 02 18:03:12.117010 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:12.116980    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:03:20.641233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.641193    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"]
Nov 02 18:03:20.645744 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.645726    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.698235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.698200    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ceb1ae-openshift-default-dockercfg-pmptc\""
Nov 02 18:03:20.713944 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.713913    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.713976    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.714032    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.714080    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.714108    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.714228    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.714269    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.714293    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.714411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.714326    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmks\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-kube-api-access-crmks\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.750623 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.750587    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"]
Nov 02 18:03:20.815002 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.814972    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815018    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815049    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815075    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815171    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815198    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815230    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815270    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmks\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-kube-api-access-crmks\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815406    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815484    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815639    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815679    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.815745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.815737    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.816298 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.816273    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.817904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.817885    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.818515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.818494    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.873925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.873888    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.874148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.874092    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmks\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-kube-api-access-crmks\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:20.959855 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:20.959822    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:21.443624 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:21.443599    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"]
Nov 02 18:03:21.445069 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:03:21.445044    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db270dc_3200_4d19_9484_c6f0c9e79cd7.slice/crio-91ad8e6f29aea4352bd129f53a3cb749e82b96e10dc1af12dd6a27a2dc498b96 WatchSource:0}: Error finding container 91ad8e6f29aea4352bd129f53a3cb749e82b96e10dc1af12dd6a27a2dc498b96: Status 404 returned error can't find the container with id 91ad8e6f29aea4352bd129f53a3cb749e82b96e10dc1af12dd6a27a2dc498b96
Nov 02 18:03:21.447555 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:21.447519    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:21.447669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:21.447601    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:21.447669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:21.447644    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:22.247831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:22.247793    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" event={"ID":"3db270dc-3200-4d19-9484-c6f0c9e79cd7","Type":"ContainerStarted","Data":"72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d"}
Nov 02 18:03:22.248373 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:22.247855    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" event={"ID":"3db270dc-3200-4d19-9484-c6f0c9e79cd7","Type":"ContainerStarted","Data":"91ad8e6f29aea4352bd129f53a3cb749e82b96e10dc1af12dd6a27a2dc498b96"}
Nov 02 18:03:22.466226 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:22.466173    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podStartSLOduration=2.466157744 podStartE2EDuration="2.466157744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:22.414726753 +0000 UTC m=+2432.671757373" watchObservedRunningTime="2026-11-02 18:03:22.466157744 +0000 UTC m=+2432.723188364"
Nov 02 18:03:22.961566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:22.961487    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:22.964019 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:22.963985    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 18:03:22.964192 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:22.964041    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 18:03:23.961012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:23.960530    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 18:03:23.961012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:23.960611    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 18:03:24.961148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:24.961083    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 18:03:24.961561 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:24.961174    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 18:03:25.960187 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:25.960148    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 18:03:25.960367 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:25.960203    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 18:03:26.961174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:26.961109    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 18:03:26.961643 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:26.961209    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 18:03:27.960664 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:27.960629    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 18:03:27.960857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:27.960685    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 18:03:29.059930 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:03:29.059893    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:03:29.408869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:29.408769    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:29.409249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:29.409229    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:29.409899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:03:29.409880    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:03:59.053146 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:03:59.053085    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:04:25.996724 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:25.996690    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"]
Nov 02 18:04:29.053156 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:04:29.053099    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:04:30.997369 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:30.997327    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n" podUID="7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" containerName="payload-processing" containerID="cri-o://656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5" gracePeriod=25
Nov 02 18:04:31.280382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.280353    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:04:31.396057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.396024    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-kube-api-access-7zzsx\" (UniqueName: \"kubernetes.io/projected/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-kube-api-access-7zzsx\") pod \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") "
Nov 02 18:04:31.396261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.396159    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-plugins-config-volume\") pod \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\" (UID: \"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934\") "
Nov 02 18:04:31.396521 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.396495    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-plugins-config-volume" pod "7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" (UID: "7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:04:31.398837 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.398809    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-kube-api-access-7zzsx" pod "7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" (UID: "7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934"). InnerVolumeSpecName "kube-api-access-7zzsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:04:31.497173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.497112    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:31.497173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.497164    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzsx\" (UniqueName: \"kubernetes.io/projected/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934-kube-api-access-7zzsx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:31.688070 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.687979    4956 generic.go:356] "Generic (PLEG): container finished" podID="7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" containerID="656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5" exitCode=0
Nov 02 18:04:31.688070 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.688029    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"
Nov 02 18:04:31.688297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.688032    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n" event={"ID":"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934","Type":"ContainerDied","Data":"656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5"}
Nov 02 18:04:31.688297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.688179    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n" event={"ID":"7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934","Type":"ContainerDied","Data":"a839fe37e8398bb623024a2f4d28cf8c4ed337240eb75b12e537b77fdb0d795b"}
Nov 02 18:04:31.688297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.688209    4956 scope.go:122] "RemoveContainer" containerID="656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5"
Nov 02 18:04:31.698715 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.698699    4956 scope.go:122] "RemoveContainer" containerID="656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5"
Nov 02 18:04:31.699010 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:04:31.698993    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5\": container with ID starting with 656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5 not found: ID does not exist" containerID="656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5"
Nov 02 18:04:31.699075 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.699018    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5"} err="failed to get container status \"656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5\": rpc error: code = NotFound desc = could not find container \"656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5\": container with ID starting with 656b467695c52da90a29bac7df75abb00435bedd6ca6c0de7ca47c39e5eb17b5 not found: ID does not exist"
Nov 02 18:04:31.852587 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.852549    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"]
Nov 02 18:04:31.902314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:31.902283    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf2b8a-75c444b649-4nj9n"]
Nov 02 18:04:33.864793 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:04:33.864760    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" path="/var/lib/kubelet/pods/7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934/volumes"
Nov 02 18:04:59.052138 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:04:59.052088    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:05:12.102291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.102249    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"]
Nov 02 18:05:12.102887 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.102509    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92" podUID="45a5089c-912e-476a-8298-e0a62e317a1b" containerName="maas-api" containerID="cri-o://6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906" gracePeriod=30
Nov 02 18:05:12.387429 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.387402    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:05:12.505975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.505672    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-metrics-certs\") pod \"45a5089c-912e-476a-8298-e0a62e317a1b\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") "
Nov 02 18:05:12.506923 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.506895    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a5089c-912e-476a-8298-e0a62e317a1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a5089c-912e-476a-8298-e0a62e317a1b-openshift-service-ca\") pod \"45a5089c-912e-476a-8298-e0a62e317a1b\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") "
Nov 02 18:05:12.507077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.506976    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-maas-api-tls\") pod \"45a5089c-912e-476a-8298-e0a62e317a1b\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") "
Nov 02 18:05:12.507077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.507011    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a5089c-912e-476a-8298-e0a62e317a1b-kube-api-access-982bf\" (UniqueName: \"kubernetes.io/projected/45a5089c-912e-476a-8298-e0a62e317a1b-kube-api-access-982bf\") pod \"45a5089c-912e-476a-8298-e0a62e317a1b\" (UID: \"45a5089c-912e-476a-8298-e0a62e317a1b\") "
Nov 02 18:05:12.507936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.507898    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a5089c-912e-476a-8298-e0a62e317a1b-openshift-service-ca" pod "45a5089c-912e-476a-8298-e0a62e317a1b" (UID: "45a5089c-912e-476a-8298-e0a62e317a1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:12.510443 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.510413    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a5089c-912e-476a-8298-e0a62e317a1b-kube-api-access-982bf" pod "45a5089c-912e-476a-8298-e0a62e317a1b" (UID: "45a5089c-912e-476a-8298-e0a62e317a1b"). InnerVolumeSpecName "kube-api-access-982bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:12.510552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.510497    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-metrics-certs" pod "45a5089c-912e-476a-8298-e0a62e317a1b" (UID: "45a5089c-912e-476a-8298-e0a62e317a1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:05:12.511622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.511579    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-maas-api-tls" pod "45a5089c-912e-476a-8298-e0a62e317a1b" (UID: "45a5089c-912e-476a-8298-e0a62e317a1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:05:12.608896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.608814    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:12.608896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.608850    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a5089c-912e-476a-8298-e0a62e317a1b-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:12.608896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.608862    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a5089c-912e-476a-8298-e0a62e317a1b-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:12.608896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.608871    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-982bf\" (UniqueName: \"kubernetes.io/projected/45a5089c-912e-476a-8298-e0a62e317a1b-kube-api-access-982bf\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:12.927418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.927325    4956 generic.go:356] "Generic (PLEG): container finished" podID="45a5089c-912e-476a-8298-e0a62e317a1b" containerID="6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906" exitCode=0
Nov 02 18:05:12.927418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.927373    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"
Nov 02 18:05:12.927418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.927409    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92" event={"ID":"45a5089c-912e-476a-8298-e0a62e317a1b","Type":"ContainerDied","Data":"6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906"}
Nov 02 18:05:12.927662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.927467    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92" event={"ID":"45a5089c-912e-476a-8298-e0a62e317a1b","Type":"ContainerDied","Data":"088ca3650704d91c92366ddcd4b3e5c1209e3d1e24960b9e38077d0ef277429d"}
Nov 02 18:05:12.927662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.927496    4956 scope.go:122] "RemoveContainer" containerID="6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906"
Nov 02 18:05:12.937559 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.937543    4956 scope.go:122] "RemoveContainer" containerID="6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906"
Nov 02 18:05:12.937825 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:05:12.937807    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906\": container with ID starting with 6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906 not found: ID does not exist" containerID="6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906"
Nov 02 18:05:12.937872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:12.937843    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906"} err="failed to get container status \"6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906\": rpc error: code = NotFound desc = could not find container \"6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906\": container with ID starting with 6e88e97d1af7877f520ed6b4d1408a5966e18f57fcb4de7bd09ee61a9a4bd906 not found: ID does not exist"
Nov 02 18:05:13.092359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:13.092315    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"]
Nov 02 18:05:13.096368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:13.096344    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cf2b8a-7d88db8d88-rzw92"]
Nov 02 18:05:13.865243 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:13.865210    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a5089c-912e-476a-8298-e0a62e317a1b" path="/var/lib/kubelet/pods/45a5089c-912e-476a-8298-e0a62e317a1b/volumes"
Nov 02 18:05:23.635027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:23.634991    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"]
Nov 02 18:05:23.635458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:23.635202    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy" containerID="cri-o://158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff" gracePeriod=30
Nov 02 18:05:28.926150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:28.926102    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:05:29.000018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:28.999982    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-envoy\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000237 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000105    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-kube-api-access-86lcs\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-kube-api-access-86lcs\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000237 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000141    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-data\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000237 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000161    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-podinfo\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000237 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000195    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-credential-socket\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000428 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000272    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-socket\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000428 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000334    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-token\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-token\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000428 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000382    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92f5015-8f4d-4c26-94d0-b30257b5183b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92f5015-8f4d-4c26-94d0-b30257b5183b-istiod-ca-cert\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000419    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-data" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:29.000608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000473    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-certs\") pod \"e92f5015-8f4d-4c26-94d0-b30257b5183b\" (UID: \"e92f5015-8f4d-4c26-94d0-b30257b5183b\") "
Nov 02 18:05:29.000608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000591    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-credential-socket" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:29.000812 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000789    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-certs" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:29.001046 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.000937    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92f5015-8f4d-4c26-94d0-b30257b5183b-istiod-ca-cert" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:29.001210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.001084    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-socket" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:29.001210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.001169    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92f5015-8f4d-4c26-94d0-b30257b5183b-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.001210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.001189    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.001210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.001205    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.001436 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.001220    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.002574 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.002547    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-envoy" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:29.002574 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.002565    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-podinfo" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:05:29.002704 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.002589    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-kube-api-access-86lcs" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "kube-api-access-86lcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:29.002842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.002827    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-token" pod "e92f5015-8f4d-4c26-94d0-b30257b5183b" (UID: "e92f5015-8f4d-4c26-94d0-b30257b5183b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:29.020113 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.020080    4956 generic.go:356] "Generic (PLEG): container finished" podID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerID="158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff" exitCode=0
Nov 02 18:05:29.020263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.020205    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"
Nov 02 18:05:29.020263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.020225    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" event={"ID":"e92f5015-8f4d-4c26-94d0-b30257b5183b","Type":"ContainerDied","Data":"158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff"}
Nov 02 18:05:29.020378 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.020276    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb" event={"ID":"e92f5015-8f4d-4c26-94d0-b30257b5183b","Type":"ContainerDied","Data":"d6f4dc72a1c6135d33fecc51c34948c3058bd5c978e26568218be21c81b9e8fc"}
Nov 02 18:05:29.020378 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.020299    4956 scope.go:122] "RemoveContainer" containerID="158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff"
Nov 02 18:05:29.031314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.031295    4956 scope.go:122] "RemoveContainer" containerID="77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d"
Nov 02 18:05:29.041956 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.041929    4956 scope.go:122] "RemoveContainer" containerID="158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff"
Nov 02 18:05:29.042244 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:05:29.042224    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff\": container with ID starting with 158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff not found: ID does not exist" containerID="158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff"
Nov 02 18:05:29.042326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.042252    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff"} err="failed to get container status \"158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff\": rpc error: code = NotFound desc = could not find container \"158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff\": container with ID starting with 158577e61d88af0f625db707ce840ff929bfc124e37e1b9c16a37ea94f6979ff not found: ID does not exist"
Nov 02 18:05:29.042326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.042269    4956 scope.go:122] "RemoveContainer" containerID="77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d"
Nov 02 18:05:29.042532 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:05:29.042507    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d\": container with ID starting with 77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d not found: ID does not exist" containerID="77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d"
Nov 02 18:05:29.042595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.042542    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d"} err="failed to get container status \"77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d\": rpc error: code = NotFound desc = could not find container \"77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d\": container with ID starting with 77ab9b38f3a5a3efb04e238efb970611b28b48e269cb768228e16b96ede7273d not found: ID does not exist"
Nov 02 18:05:29.053878 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:05:29.053850    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:05:29.102738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.102708    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86lcs\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-kube-api-access-86lcs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.102738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.102735    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.102738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.102746    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.102965 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.102754    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.102965 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.102766    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92f5015-8f4d-4c26-94d0-b30257b5183b-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:29.209731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.209704    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"]
Nov 02 18:05:29.217456 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.217425    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cf2b8a-openshift-default-6fb7df8b85-2ssmb"]
Nov 02 18:05:29.865196 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:29.865166    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" path="/var/lib/kubelet/pods/e92f5015-8f4d-4c26-94d0-b30257b5183b/volumes"
Nov 02 18:05:36.673834 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.673801    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"]
Nov 02 18:05:36.674336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674322    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy"
Nov 02 18:05:36.674382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674338    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy"
Nov 02 18:05:36.674382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674359    4956 cpu_manager.go:414] "containerMap: removing container" podUID="7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" containerName="payload-processing"
Nov 02 18:05:36.674382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674365    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" containerName="payload-processing"
Nov 02 18:05:36.674382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674376    4956 cpu_manager.go:414] "containerMap: removing container" podUID="45a5089c-912e-476a-8298-e0a62e317a1b" containerName="maas-api"
Nov 02 18:05:36.674382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674381    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a5089c-912e-476a-8298-e0a62e317a1b" containerName="maas-api"
Nov 02 18:05:36.674545 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674485    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy"
Nov 02 18:05:36.674545 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674497    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a17c2ff-3427-4c72-bfcf-d4f2ea9bf934" containerName="payload-processing"
Nov 02 18:05:36.674545 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.674506    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a5089c-912e-476a-8298-e0a62e317a1b" containerName="maas-api"
Nov 02 18:05:36.679075 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.679039    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.723433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.723399    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"]
Nov 02 18:05:36.724041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.724025    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy"
Nov 02 18:05:36.724107 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.724043    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy"
Nov 02 18:05:36.724205 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.724191    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92f5015-8f4d-4c26-94d0-b30257b5183b" containerName="istio-proxy"
Nov 02 18:05:36.727439 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.727421    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.774547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.774511    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"]
Nov 02 18:05:36.787958 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.787928    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"]
Nov 02 18:05:36.791259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791228    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.791425 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791279    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llb6c\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-kube-api-access-llb6c\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.791483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791417    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.791483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791469    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.791559 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791502    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.791559 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791528    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.791639 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791557    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.791639 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791592    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.791712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791674    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.791712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791697    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.791796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791731    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.791796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791782    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.792823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791818    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.792823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791850    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9t6f\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-kube-api-access-n9t6f\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.792823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791883    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.792823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791924    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.792823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.791949    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.792823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.792386    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.893396 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893357    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.893598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893404    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.893598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893429    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.893598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893458    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.893598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893493    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.893598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893573    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.893598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893598    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893633    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893680    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893718    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893750    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9t6f\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-kube-api-access-n9t6f\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893787    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893830    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893836    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893857    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893916    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893963    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894006    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894031    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llb6c\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-kube-api-access-llb6c\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894078    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.894338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894086    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.895073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894422    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.895073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.893846    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.895073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894458    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.895073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894702    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.895073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894823    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.895073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894927    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.895073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.894999    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.896479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.896456    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.896784 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.896753    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.896942 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.896923    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.897006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.896975    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.951736 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.951695    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.951736 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.951734    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llb6c\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-kube-api-access-llb6c\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.951930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.951748    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:36.951930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.951776    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9t6f\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-kube-api-access-n9t6f\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:36.993060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:36.993022    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:37.042592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.042547    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:37.427100 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.426186    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"]
Nov 02 18:05:37.459888 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:05:37.459855    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c4aa2e_ef7c_4dd0_b39f_5ac417993b01.slice/crio-280e025dc20568864933c958c579f3afe2c8436411827fe51ded53d3905ecbcc WatchSource:0}: Error finding container 280e025dc20568864933c958c579f3afe2c8436411827fe51ded53d3905ecbcc: Status 404 returned error can't find the container with id 280e025dc20568864933c958c579f3afe2c8436411827fe51ded53d3905ecbcc
Nov 02 18:05:37.462214 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.462181    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:37.462328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.462247    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:37.462328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.462276    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:37.473041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.473018    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"]
Nov 02 18:05:37.474063 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:05:37.474040    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3572bea_8ee7_4bf6_8623_65372e8ad09f.slice/crio-a031fd1871423144962bd9f43f0250d7cf9b1412ef557365514ffdbf7b99cbb9 WatchSource:0}: Error finding container a031fd1871423144962bd9f43f0250d7cf9b1412ef557365514ffdbf7b99cbb9: Status 404 returned error can't find the container with id a031fd1871423144962bd9f43f0250d7cf9b1412ef557365514ffdbf7b99cbb9
Nov 02 18:05:37.476458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.476428    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:37.476563 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.476492    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:37.476563 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:37.476520    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:38.082840 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.082799    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" event={"ID":"c3572bea-8ee7-4bf6-8623-65372e8ad09f","Type":"ContainerStarted","Data":"235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797"}
Nov 02 18:05:38.083384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.082863    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" event={"ID":"c3572bea-8ee7-4bf6-8623-65372e8ad09f","Type":"ContainerStarted","Data":"a031fd1871423144962bd9f43f0250d7cf9b1412ef557365514ffdbf7b99cbb9"}
Nov 02 18:05:38.084754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.084726    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" event={"ID":"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01","Type":"ContainerStarted","Data":"8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc"}
Nov 02 18:05:38.084851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.084779    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" event={"ID":"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01","Type":"ContainerStarted","Data":"280e025dc20568864933c958c579f3afe2c8436411827fe51ded53d3905ecbcc"}
Nov 02 18:05:38.385506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.385407    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podStartSLOduration=2.385390736 podStartE2EDuration="2.385390736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:38.383783382 +0000 UTC m=+2568.640814003" watchObservedRunningTime="2026-11-02 18:05:38.385390736 +0000 UTC m=+2568.642421356"
Nov 02 18:05:38.385814 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.385794    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podStartSLOduration=2.385786532 podStartE2EDuration="2.385786532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:38.248857593 +0000 UTC m=+2568.505888215" watchObservedRunningTime="2026-11-02 18:05:38.385786532 +0000 UTC m=+2568.642817148"
Nov 02 18:05:38.993650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.993607    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:38.995098 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.995071    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 18:05:38.995230 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:38.995151    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 18:05:39.047804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:39.047764    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:39.064147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:39.061268    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 18:05:39.064147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:39.061340    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 18:05:39.993983 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:39.993944    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 18:05:39.994470 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:39.993997    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 18:05:40.075165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:40.074316    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 18:05:40.075165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:40.074387    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 18:05:40.993705 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:40.993663    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 18:05:40.993872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:40.993723    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 18:05:41.047144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:41.046587    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 18:05:41.047144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:41.046653    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 18:05:41.993662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:41.993595    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 18:05:41.993662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:41.993644    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 18:05:42.050156 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:42.049913    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 18:05:42.050156 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:42.049977    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 18:05:42.993722 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:42.993687    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 18:05:42.993993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:42.993742    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 18:05:43.048160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:43.047565    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 18:05:43.048160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:43.047629    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 18:05:43.993803 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:43.993761    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 18:05:43.994221 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:43.993849    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 18:05:44.043496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:44.043456    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 18:05:44.043691 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:44.043507    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 18:05:44.998288 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:44.998260    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:44.998727 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:44.998708    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:44.999433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:44.999413    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:05:45.047729 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:45.047701    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:45.047984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:45.047968    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:45.048790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:05:45.048771    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:05:59.055244 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:05:59.055205    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:06:08.341110 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.341064    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"]
Nov 02 18:06:08.345218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.345190    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.397299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.397261    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\""
Nov 02 18:06:08.450435 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.450398    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"]
Nov 02 18:06:08.475875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.475837    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476049 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.475894    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476049 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.475963    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476139 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.476062    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5klk\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-kube-api-access-z5klk\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476178 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.476160    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476221 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.476211    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476276 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.476236    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476325 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.476315    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.476387 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.476369    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577447 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577409    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577466    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577496    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577517    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5klk\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-kube-api-access-z5klk\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577545    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577594    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577617    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.577662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577642    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.578036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577726    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.578036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.577962    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.578182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.578082    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.578228 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.578192    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.578268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.578220    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.579098 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.579061    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.580913 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.580883    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.589635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.589608    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.635227 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.635156    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.635419 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.635395    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5klk\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-kube-api-access-z5klk\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:08.659172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:08.659139    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:09.089560 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:06:09.089530    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8be0887_a6f6_4b19_8272_ec68a0c1b17d.slice/crio-19c3bfcab0a401cd3260525784c635bd2aedb39350da3b38bd6cc4543b02a1f8 WatchSource:0}: Error finding container 19c3bfcab0a401cd3260525784c635bd2aedb39350da3b38bd6cc4543b02a1f8: Status 404 returned error can't find the container with id 19c3bfcab0a401cd3260525784c635bd2aedb39350da3b38bd6cc4543b02a1f8
Nov 02 18:06:09.092035 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.092004    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:06:09.092103 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.092074    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:06:09.092176 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.092101    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:06:09.148372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.145554    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"]
Nov 02 18:06:09.311176 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.311139    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" event={"ID":"e8be0887-a6f6-4b19-8272-ec68a0c1b17d","Type":"ContainerStarted","Data":"7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061"}
Nov 02 18:06:09.311387 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.311206    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" event={"ID":"e8be0887-a6f6-4b19-8272-ec68a0c1b17d","Type":"ContainerStarted","Data":"19c3bfcab0a401cd3260525784c635bd2aedb39350da3b38bd6cc4543b02a1f8"}
Nov 02 18:06:09.431966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.431857    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podStartSLOduration=1.431842716 podStartE2EDuration="1.431842716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:09.428264246 +0000 UTC m=+2599.685294865" watchObservedRunningTime="2026-11-02 18:06:09.431842716 +0000 UTC m=+2599.688873338"
Nov 02 18:06:09.659428 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:09.659397    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:10.664150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:10.661712    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 18:06:10.664150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:10.661779    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 18:06:11.661150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:11.660246    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 18:06:11.661150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:11.660308    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 18:06:12.247304 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:12.247233    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:06:12.278019 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:12.277972    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 18:06:12.507603 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:12.507185    4956 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:06:12.660629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:12.660587    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 18:06:12.660841 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:12.660655    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 18:06:13.659790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:13.659756    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 18:06:13.660198 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:13.659824    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 18:06:14.659842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:14.659802    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 18:06:14.660390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:14.659874    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 18:06:15.660451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:15.660404    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 18:06:15.660889 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:15.660488    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 18:06:17.700287 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:17.700248    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:06:17.700746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:17.700305    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:06:17.787459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:17.787418    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:17.788083 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:17.788024    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:17.789027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:17.789005    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:06:17.842296 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:17.842273    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gtcjl_8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44/payload-processing/0.log"
Nov 02 18:06:18.078335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:18.078246    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gtcjl_8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44/payload-processing/0.log"
Nov 02 18:06:23.531576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.531536    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"]
Nov 02 18:06:23.535771 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.535751    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.642974 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.642939    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"]
Nov 02 18:06:23.643182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643150    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643235    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643370 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643355    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftvj\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-kube-api-access-mftvj\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643385    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643580 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643561    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5094140-37a0-4338-8033-be2df9a9a3c8-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643636 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643621    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643692    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643801 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643781    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.643863 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.643827    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.746066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.746023    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.746739 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.746715    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.747355 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.747339    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.747787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.747773    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.753974    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.754028    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.754144    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftvj\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-kube-api-access-mftvj\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.754173    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.754362    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5094140-37a0-4338-8033-be2df9a9a3c8-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.747705    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.755020    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.747269    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.758604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.755245    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.759200 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.758926    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.759609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.759582    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5094140-37a0-4338-8033-be2df9a9a3c8-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.763755 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.763723    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.804041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.803956    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.812622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.812593    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftvj\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-kube-api-access-mftvj\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:23.850672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:23.850633    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:24.281454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.281428    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"]
Nov 02 18:06:24.283035 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:06:24.283007    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5094140_37a0_4338_8033_be2df9a9a3c8.slice/crio-20c39d4ca5a46e85178381d9dfa5a3f467e3ab9525d6ee0be2a5d9edf864a3a5 WatchSource:0}: Error finding container 20c39d4ca5a46e85178381d9dfa5a3f467e3ab9525d6ee0be2a5d9edf864a3a5: Status 404 returned error can't find the container with id 20c39d4ca5a46e85178381d9dfa5a3f467e3ab9525d6ee0be2a5d9edf864a3a5
Nov 02 18:06:24.285626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.285588    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:06:24.285716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.285676    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:06:24.285754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.285721    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:06:24.569769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.569673    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" event={"ID":"b5094140-37a0-4338-8033-be2df9a9a3c8","Type":"ContainerStarted","Data":"b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4"}
Nov 02 18:06:24.569769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.569732    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" event={"ID":"b5094140-37a0-4338-8033-be2df9a9a3c8","Type":"ContainerStarted","Data":"20c39d4ca5a46e85178381d9dfa5a3f467e3ab9525d6ee0be2a5d9edf864a3a5"}
Nov 02 18:06:24.691006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.690949    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podStartSLOduration=1.690933918 podStartE2EDuration="1.690933918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:24.688475299 +0000 UTC m=+2614.945505920" watchObservedRunningTime="2026-11-02 18:06:24.690933918 +0000 UTC m=+2614.947964537"
Nov 02 18:06:24.851183 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:24.851092    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:25.852941 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:25.852894    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 18:06:25.853503 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:25.852951    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 18:06:26.851340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:26.851302    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 18:06:26.851569 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:26.851361    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 18:06:27.851653 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:27.851615    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 18:06:27.852107 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:27.851673    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 18:06:28.851884 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:28.851848    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 18:06:28.852347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:28.851904    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 18:06:29.077270 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:06:29.077233    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:06:29.851712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:29.851670    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 18:06:29.851911 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:29.851738    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 18:06:30.851923 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:30.851882    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 18:06:30.852358 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:30.851965    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 18:06:31.856150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:31.856084    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:31.856548 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:31.856512    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:31.864487 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:31.864457    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:06:32.839955 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:32.839918    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"]
Nov 02 18:06:32.844037 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:32.844021    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:32.897220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:32.897188    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f7aa2e\""
Nov 02 18:06:32.897722 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:32.897531    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f7aa2e-dockercfg-qz6np\""
Nov 02 18:06:32.902300 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:32.902273    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"]
Nov 02 18:06:32.997082 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:32.997042    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea85e4-0611-4640-8172-95584a79424e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:32.997327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:32.997205    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4lf\" (UniqueName: \"kubernetes.io/projected/49ea85e4-0611-4640-8172-95584a79424e-kube-api-access-vr4lf\") pod \"payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:33.098026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:33.097927    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea85e4-0611-4640-8172-95584a79424e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:33.098026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:33.097997    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4lf\" (UniqueName: \"kubernetes.io/projected/49ea85e4-0611-4640-8172-95584a79424e-kube-api-access-vr4lf\") pod \"payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:33.098673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:33.098650    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea85e4-0611-4640-8172-95584a79424e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:33.159785 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:33.159736    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4lf\" (UniqueName: \"kubernetes.io/projected/49ea85e4-0611-4640-8172-95584a79424e-kube-api-access-vr4lf\") pod \"payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:33.458544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:33.458510    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:33.838202 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:33.838173    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"]
Nov 02 18:06:33.839660 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:06:33.839630    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea85e4_0611_4640_8172_95584a79424e.slice/crio-6da0a96cbf5d0bcf7fd70e818152ac0a59c6925040f5516a6ffda4380d92a927 WatchSource:0}: Error finding container 6da0a96cbf5d0bcf7fd70e818152ac0a59c6925040f5516a6ffda4380d92a927: Status 404 returned error can't find the container with id 6da0a96cbf5d0bcf7fd70e818152ac0a59c6925040f5516a6ffda4380d92a927
Nov 02 18:06:34.638956 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:34.638917    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" event={"ID":"49ea85e4-0611-4640-8172-95584a79424e","Type":"ContainerStarted","Data":"53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788"}
Nov 02 18:06:34.639398 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:34.638979    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" event={"ID":"49ea85e4-0611-4640-8172-95584a79424e","Type":"ContainerStarted","Data":"6da0a96cbf5d0bcf7fd70e818152ac0a59c6925040f5516a6ffda4380d92a927"}
Nov 02 18:06:34.639398 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:34.639005    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:34.760796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:34.760738    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" podStartSLOduration=2.760725907 podStartE2EDuration="2.760725907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:34.75703074 +0000 UTC m=+2625.014061361" watchObservedRunningTime="2026-11-02 18:06:34.760725907 +0000 UTC m=+2625.017756527"
Nov 02 18:06:37.717201 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.714212    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"]
Nov 02 18:06:37.719643 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.719616    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:37.834341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.831437    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-413a4e-dockercfg-smt8h\""
Nov 02 18:06:37.834341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.832210    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-413a4e\""
Nov 02 18:06:37.861914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.846920    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"]
Nov 02 18:06:37.861914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.848299    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmwf7\" (UniqueName: \"kubernetes.io/projected/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-kube-api-access-vmwf7\") pod \"payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:37.861914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.848393    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:37.954144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.952002    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmwf7\" (UniqueName: \"kubernetes.io/projected/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-kube-api-access-vmwf7\") pod \"payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:37.954144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.952134    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:37.954144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:37.953088    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:38.019917 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:38.019839    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmwf7\" (UniqueName: \"kubernetes.io/projected/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-kube-api-access-vmwf7\") pod \"payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:38.249146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:38.234701    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:38.748222 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:38.748178    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"]
Nov 02 18:06:39.731515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:39.731474    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" event={"ID":"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa","Type":"ContainerStarted","Data":"ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd"}
Nov 02 18:06:39.732044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:39.731573    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" event={"ID":"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa","Type":"ContainerStarted","Data":"e264f164162d90a77614cfeb93de5388f2ff1512506a596bb03dd0326881ce64"}
Nov 02 18:06:39.732044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:39.731604    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:39.849319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:39.849251    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" podStartSLOduration=2.849235528 podStartE2EDuration="2.849235528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:39.846309159 +0000 UTC m=+2630.103339774" watchObservedRunningTime="2026-11-02 18:06:39.849235528 +0000 UTC m=+2630.106266148"
Nov 02 18:06:45.649383 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:45.649355    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:06:50.741398 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:06:50.741370    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:06:59.053493 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:06:59.053456    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:07:03.689287 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.689203    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"]
Nov 02 18:07:03.693262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.693245    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753197 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753159    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84rl\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-kube-api-access-t84rl\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753203    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753226    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753278    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753396    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753647 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753499    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753647 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753534    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753647 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753566    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.753798 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.753691    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.854781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.854742    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.854781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.854787    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.854818    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.854915    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855040    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84rl\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-kube-api-access-t84rl\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855080    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855115    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855155    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855213    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855246    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855274    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855387    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855852 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855828    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.855977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.855955    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.861394 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.861367    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.861640 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.861618    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.894692 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.894658    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"]
Nov 02 18:07:03.912335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.912301    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:03.912525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:03.912418    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84rl\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-kube-api-access-t84rl\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:04.006775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:04.006737    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:04.441143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:04.441035    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"]
Nov 02 18:07:04.471616 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:07:04.471589    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb912ddaa_0dc7_4ebd_9afb_f188eae3e1a4.slice/crio-7ec940327baaebc8ddbfdcf6806887d1a59ad3aa83a0dd64bed52d9f245fdc3d WatchSource:0}: Error finding container 7ec940327baaebc8ddbfdcf6806887d1a59ad3aa83a0dd64bed52d9f245fdc3d: Status 404 returned error can't find the container with id 7ec940327baaebc8ddbfdcf6806887d1a59ad3aa83a0dd64bed52d9f245fdc3d
Nov 02 18:07:04.474180 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:04.474147    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:04.474261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:04.474220    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:04.474261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:04.474250    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:04.892015 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:04.891976    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" event={"ID":"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4","Type":"ContainerStarted","Data":"36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a"}
Nov 02 18:07:04.892466 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:04.892047    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" event={"ID":"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4","Type":"ContainerStarted","Data":"7ec940327baaebc8ddbfdcf6806887d1a59ad3aa83a0dd64bed52d9f245fdc3d"}
Nov 02 18:07:05.007080 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.007052    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:05.008882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.008771    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podStartSLOduration=2.008755192 podStartE2EDuration="2.008755192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:05.008525319 +0000 UTC m=+2655.265555952" watchObservedRunningTime="2026-11-02 18:07:05.008755192 +0000 UTC m=+2655.265785797"
Nov 02 18:07:05.009159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.009026    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 18:07:05.009159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.009140    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 18:07:05.396549 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.396507    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"]
Nov 02 18:07:05.396767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.396722    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr" podUID="698f8516-39a5-464f-93ed-7f2aa1bbdaee" containerName="maas-api" containerID="cri-o://249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7" gracePeriod=30
Nov 02 18:07:05.754336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.749425    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:07:05.777616 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.777583    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698f8516-39a5-464f-93ed-7f2aa1bbdaee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698f8516-39a5-464f-93ed-7f2aa1bbdaee-openshift-service-ca\") pod \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") "
Nov 02 18:07:05.777788 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.777678    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698f8516-39a5-464f-93ed-7f2aa1bbdaee-kube-api-access-fzgp7\" (UniqueName: \"kubernetes.io/projected/698f8516-39a5-464f-93ed-7f2aa1bbdaee-kube-api-access-fzgp7\") pod \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") "
Nov 02 18:07:05.777788 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.777769    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls\") pod \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") "
Nov 02 18:07:05.777906 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.777826    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-metrics-certs\") pod \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\" (UID: \"698f8516-39a5-464f-93ed-7f2aa1bbdaee\") "
Nov 02 18:07:05.777988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.777962    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698f8516-39a5-464f-93ed-7f2aa1bbdaee-openshift-service-ca" pod "698f8516-39a5-464f-93ed-7f2aa1bbdaee" (UID: "698f8516-39a5-464f-93ed-7f2aa1bbdaee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:05.778327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.778304    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698f8516-39a5-464f-93ed-7f2aa1bbdaee-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:05.780073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.780047    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698f8516-39a5-464f-93ed-7f2aa1bbdaee-kube-api-access-fzgp7" pod "698f8516-39a5-464f-93ed-7f2aa1bbdaee" (UID: "698f8516-39a5-464f-93ed-7f2aa1bbdaee"). InnerVolumeSpecName "kube-api-access-fzgp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:05.780206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.780095    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls" pod "698f8516-39a5-464f-93ed-7f2aa1bbdaee" (UID: "698f8516-39a5-464f-93ed-7f2aa1bbdaee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:07:05.780287 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.780268    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-metrics-certs" pod "698f8516-39a5-464f-93ed-7f2aa1bbdaee" (UID: "698f8516-39a5-464f-93ed-7f2aa1bbdaee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:07:05.880452 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.880410    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgp7\" (UniqueName: \"kubernetes.io/projected/698f8516-39a5-464f-93ed-7f2aa1bbdaee-kube-api-access-fzgp7\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:05.880452 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.880442    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:05.880452 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.880457    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698f8516-39a5-464f-93ed-7f2aa1bbdaee-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:05.906137 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.906078    4956 generic.go:356] "Generic (PLEG): container finished" podID="698f8516-39a5-464f-93ed-7f2aa1bbdaee" containerID="249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7" exitCode=0
Nov 02 18:07:05.906588 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.906214    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr" event={"ID":"698f8516-39a5-464f-93ed-7f2aa1bbdaee","Type":"ContainerDied","Data":"249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7"}
Nov 02 18:07:05.906588 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.906268    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr" event={"ID":"698f8516-39a5-464f-93ed-7f2aa1bbdaee","Type":"ContainerDied","Data":"2db4a08d3c465e14ecc0ef6861b0d161c9e49223ff486ccacf69eb747eab2ead"}
Nov 02 18:07:05.906588 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.906297    4956 scope.go:122] "RemoveContainer" containerID="249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7"
Nov 02 18:07:05.906588 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.906489    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"
Nov 02 18:07:05.911332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.911305    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"]
Nov 02 18:07:05.911503 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.911481    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x" podUID="a57141d2-0cd9-44b4-b541-813770d37e9f" containerName="payload-processing" containerID="cri-o://26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd" gracePeriod=30
Nov 02 18:07:05.924320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.924292    4956 scope.go:122] "RemoveContainer" containerID="249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7"
Nov 02 18:07:05.924710 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:07:05.924686    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7\": container with ID starting with 249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7 not found: ID does not exist" containerID="249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7"
Nov 02 18:07:05.924838 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:05.924719    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7"} err="failed to get container status \"249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7\": rpc error: code = NotFound desc = could not find container \"249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7\": container with ID starting with 249744e1b17903abf40ba910ab0d6504dd66ec99618cd6d981576e713f3dc5a7 not found: ID does not exist"
Nov 02 18:07:06.007942 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.007913    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 18:07:06.008103 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.007997    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 18:07:06.078151 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.074733    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"]
Nov 02 18:07:06.078508 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.078486    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f07b5b-74869ccbfc-89sdr"]
Nov 02 18:07:06.290645 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.290618    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:07:06.385925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.385876    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57141d2-0cd9-44b4-b541-813770d37e9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57141d2-0cd9-44b4-b541-813770d37e9f-plugins-config-volume\") pod \"a57141d2-0cd9-44b4-b541-813770d37e9f\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") "
Nov 02 18:07:06.386221 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.386196    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57141d2-0cd9-44b4-b541-813770d37e9f-kube-api-access-tfqdx\" (UniqueName: \"kubernetes.io/projected/a57141d2-0cd9-44b4-b541-813770d37e9f-kube-api-access-tfqdx\") pod \"a57141d2-0cd9-44b4-b541-813770d37e9f\" (UID: \"a57141d2-0cd9-44b4-b541-813770d37e9f\") "
Nov 02 18:07:06.387061 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.387033    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57141d2-0cd9-44b4-b541-813770d37e9f-plugins-config-volume" pod "a57141d2-0cd9-44b4-b541-813770d37e9f" (UID: "a57141d2-0cd9-44b4-b541-813770d37e9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:06.389440 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.389390    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57141d2-0cd9-44b4-b541-813770d37e9f-kube-api-access-tfqdx" pod "a57141d2-0cd9-44b4-b541-813770d37e9f" (UID: "a57141d2-0cd9-44b4-b541-813770d37e9f"). InnerVolumeSpecName "kube-api-access-tfqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:06.493577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.493538    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqdx\" (UniqueName: \"kubernetes.io/projected/a57141d2-0cd9-44b4-b541-813770d37e9f-kube-api-access-tfqdx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:06.493577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.493580    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57141d2-0cd9-44b4-b541-813770d37e9f-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:06.913854 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.913816    4956 generic.go:356] "Generic (PLEG): container finished" podID="a57141d2-0cd9-44b4-b541-813770d37e9f" containerID="26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd" exitCode=0
Nov 02 18:07:06.914337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.913871    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"
Nov 02 18:07:06.914337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.913908    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x" event={"ID":"a57141d2-0cd9-44b4-b541-813770d37e9f","Type":"ContainerDied","Data":"26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd"}
Nov 02 18:07:06.914337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.913965    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x" event={"ID":"a57141d2-0cd9-44b4-b541-813770d37e9f","Type":"ContainerDied","Data":"f0fad9962b2556a6223bdfc9b104719b1f7e7cb5745659f180c6523c2bb3bb04"}
Nov 02 18:07:06.914337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.913995    4956 scope.go:122] "RemoveContainer" containerID="26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd"
Nov 02 18:07:06.924577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.924553    4956 scope.go:122] "RemoveContainer" containerID="26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd"
Nov 02 18:07:06.924859 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:07:06.924836    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd\": container with ID starting with 26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd not found: ID does not exist" containerID="26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd"
Nov 02 18:07:06.924925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:06.924871    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd"} err="failed to get container status \"26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd\": rpc error: code = NotFound desc = could not find container \"26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd\": container with ID starting with 26509cb3ebb39ea10f5be98630e3a7d46b04d1dda6d99730a869c6c8398818dd not found: ID does not exist"
Nov 02 18:07:07.007147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:07.007093    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 18:07:07.007286 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:07.007179    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 18:07:07.091200 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:07.091163    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"]
Nov 02 18:07:07.121300 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:07.121268    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-75bbcfb9f4-zfm8x"]
Nov 02 18:07:07.866154 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:07.866097    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698f8516-39a5-464f-93ed-7f2aa1bbdaee" path="/var/lib/kubelet/pods/698f8516-39a5-464f-93ed-7f2aa1bbdaee/volumes"
Nov 02 18:07:07.866553 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:07.866538    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57141d2-0cd9-44b4-b541-813770d37e9f" path="/var/lib/kubelet/pods/a57141d2-0cd9-44b4-b541-813770d37e9f/volumes"
Nov 02 18:07:08.007317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:08.007279    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 18:07:08.007787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:08.007345    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 18:07:09.007609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:09.007572    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 18:07:09.008010 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:09.007637    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 18:07:10.007657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:10.007616    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 18:07:10.008065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:10.007678    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 18:07:11.007178 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:11.007134    4956 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 18:07:11.007366 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:11.007199    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 18:07:12.187848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:12.187813    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:12.188378 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:12.188227    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:12.189377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:12.189355    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:07:27.564008 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.563970    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"]
Nov 02 18:07:27.564650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.564559    4956 cpu_manager.go:414] "containerMap: removing container" podUID="698f8516-39a5-464f-93ed-7f2aa1bbdaee" containerName="maas-api"
Nov 02 18:07:27.564650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.564572    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698f8516-39a5-464f-93ed-7f2aa1bbdaee" containerName="maas-api"
Nov 02 18:07:27.564650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.564586    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a57141d2-0cd9-44b4-b541-813770d37e9f" containerName="payload-processing"
Nov 02 18:07:27.564650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.564592    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57141d2-0cd9-44b4-b541-813770d37e9f" containerName="payload-processing"
Nov 02 18:07:27.564863 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.564702    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="698f8516-39a5-464f-93ed-7f2aa1bbdaee" containerName="maas-api"
Nov 02 18:07:27.564863 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.564713    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57141d2-0cd9-44b4-b541-813770d37e9f" containerName="payload-processing"
Nov 02 18:07:27.568253 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.568231    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.620623 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.620596    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-413a4e-serving-cert\""
Nov 02 18:07:27.673524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.673487    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"]
Nov 02 18:07:27.680389 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.680363    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.680523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.680414    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.680523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.680435    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.680615 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.680587    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzvd\" (UniqueName: \"kubernetes.io/projected/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-kube-api-access-lrzvd\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.781886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.781847    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.782068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.781920    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.782068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.781949    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.782068 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:07:27.782011    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-serving-cert: secret "maas-api-e2e-shared-a-w2-413a4e-serving-cert" not found
Nov 02 18:07:27.782068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.782002    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzvd\" (UniqueName: \"kubernetes.io/projected/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-kube-api-access-lrzvd\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.782330 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:07:27.782091    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls podName:2fced4cc-74fe-4694-83a3-d3b2a80a8acb nodeName:}" failed. No retries permitted until 2026-11-02 18:07:28.282070076 +0000 UTC m=+2678.539100675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls") pod "maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd" (UID: "2fced4cc-74fe-4694-83a3-d3b2a80a8acb") : secret "maas-api-e2e-shared-a-w2-413a4e-serving-cert" not found
Nov 02 18:07:27.782754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.782718    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.784647 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.784628    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:27.841137 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:27.841038    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzvd\" (UniqueName: \"kubernetes.io/projected/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-kube-api-access-lrzvd\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:28.288137 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:28.288093    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:28.290719 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:28.290694    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:28.482393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:28.482348    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:28.873858 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:28.873824    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"]
Nov 02 18:07:28.880959 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:07:28.880932    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fced4cc_74fe_4694_83a3_d3b2a80a8acb.slice/crio-0b65a06cb0e51ec1c01bc3b1b944265f8f67e9dd5c0fba021c46dff8fe548088 WatchSource:0}: Error finding container 0b65a06cb0e51ec1c01bc3b1b944265f8f67e9dd5c0fba021c46dff8fe548088: Status 404 returned error can't find the container with id 0b65a06cb0e51ec1c01bc3b1b944265f8f67e9dd5c0fba021c46dff8fe548088
Nov 02 18:07:28.882771 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:28.882751    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:07:29.053554 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:07:29.053518    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:07:29.072554 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:29.072515    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd" event={"ID":"2fced4cc-74fe-4694-83a3-d3b2a80a8acb","Type":"ContainerStarted","Data":"0b65a06cb0e51ec1c01bc3b1b944265f8f67e9dd5c0fba021c46dff8fe548088"}
Nov 02 18:07:30.080359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:30.080318    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd" event={"ID":"2fced4cc-74fe-4694-83a3-d3b2a80a8acb","Type":"ContainerStarted","Data":"0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c"}
Nov 02 18:07:30.080760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:30.080386    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:30.205263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:30.205199    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd" podStartSLOduration=2.408277812 podStartE2EDuration="3.205185768s" totalImagesPullingTime="796.907956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:30.20216737 +0000 UTC m=+2680.459197996" watchObservedRunningTime="2026-11-02 18:07:30.205185768 +0000 UTC m=+2680.462216387"
Nov 02 18:07:36.093048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:36.093013    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:07:50.211597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:50.211568    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:07:50.213441 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:50.213418    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:07:53.942897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:53.942829    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"]
Nov 02 18:07:53.953239 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:53.953199    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.005745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.005707    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"]
Nov 02 18:07:54.006552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.006518    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec782aa8-8d0f-4822-840a-da1d25dfae8f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.006721 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.006694    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65c9\" (UniqueName: \"kubernetes.io/projected/ec782aa8-8d0f-4822-840a-da1d25dfae8f-kube-api-access-c65c9\") pod \"payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.051514 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.051478    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-eb1633\""
Nov 02 18:07:54.051514 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.051489    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-eb1633-dockercfg-vpd64\""
Nov 02 18:07:54.107935 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.107905    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec782aa8-8d0f-4822-840a-da1d25dfae8f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.108136 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.108035    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65c9\" (UniqueName: \"kubernetes.io/projected/ec782aa8-8d0f-4822-840a-da1d25dfae8f-kube-api-access-c65c9\") pod \"payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.108687 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.108666    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec782aa8-8d0f-4822-840a-da1d25dfae8f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.165554 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.165517    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65c9\" (UniqueName: \"kubernetes.io/projected/ec782aa8-8d0f-4822-840a-da1d25dfae8f-kube-api-access-c65c9\") pod \"payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.292617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.292585    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:54.669591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:54.669556    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"]
Nov 02 18:07:54.670887 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:07:54.670863    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec782aa8_8d0f_4822_840a_da1d25dfae8f.slice/crio-7e464ecce75727fb93be34119e7cbcffe3d08fcc26cc6b6a71f087398f65c618 WatchSource:0}: Error finding container 7e464ecce75727fb93be34119e7cbcffe3d08fcc26cc6b6a71f087398f65c618: Status 404 returned error can't find the container with id 7e464ecce75727fb93be34119e7cbcffe3d08fcc26cc6b6a71f087398f65c618
Nov 02 18:07:55.240086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:55.240048    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb" event={"ID":"ec782aa8-8d0f-4822-840a-da1d25dfae8f","Type":"ContainerStarted","Data":"4f8f5f35e24f86166d7e1e2907481c0fd04fe61b149b5bb15eea2d79d303cbaf"}
Nov 02 18:07:55.240519 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:55.240110    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb" event={"ID":"ec782aa8-8d0f-4822-840a-da1d25dfae8f","Type":"ContainerStarted","Data":"7e464ecce75727fb93be34119e7cbcffe3d08fcc26cc6b6a71f087398f65c618"}
Nov 02 18:07:55.240519 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:55.240262    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:07:55.357505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:07:55.357446    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb" podStartSLOduration=2.357433158 podStartE2EDuration="2.357433158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:55.354529723 +0000 UTC m=+2705.611560343" watchObservedRunningTime="2026-11-02 18:07:55.357433158 +0000 UTC m=+2705.614463829"
Nov 02 18:07:59.053901 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:07:59.053861    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:08:06.248711 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:06.248678    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:08:21.595557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.595524    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"]
Nov 02 18:08:21.599642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.599624    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.656055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.656015    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"]
Nov 02 18:08:21.665628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.665591    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsgn7\" (UniqueName: \"kubernetes.io/projected/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-kube-api-access-hsgn7\") pod \"payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.666170 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.666147    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.732146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.707420    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-0ad26a\""
Nov 02 18:08:21.734362 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.733667    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-0ad26a-dockercfg-m4r2c\""
Nov 02 18:08:21.767760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.767717    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgn7\" (UniqueName: \"kubernetes.io/projected/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-kube-api-access-hsgn7\") pod \"payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.767981 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.767811    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.768786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.768756    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.774679 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.774652    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"]
Nov 02 18:08:21.779080 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.779057    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:21.820658 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.820626    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"]
Nov 02 18:08:21.825641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.825601    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgn7\" (UniqueName: \"kubernetes.io/projected/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-kube-api-access-hsgn7\") pod \"payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.868731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.868652    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:21.868731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.868714    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5jz\" (UniqueName: \"kubernetes.io/projected/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-kube-api-access-lb5jz\") pod \"payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:21.913299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.913268    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:21.969921 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.969886    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:21.970065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.969983    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5jz\" (UniqueName: \"kubernetes.io/projected/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-kube-api-access-lb5jz\") pod \"payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:21.971330 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:21.970780    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:22.037652 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.036646    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5jz\" (UniqueName: \"kubernetes.io/projected/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-kube-api-access-lb5jz\") pod \"payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:22.102926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.102705    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:22.299385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.299352    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"]
Nov 02 18:08:22.299680 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:08:22.299651    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8045263d_4fe6_40b7_b83d_31ecc6c5f7cc.slice/crio-0294bd11cbff9a52a082aa1b3fba33fd68829086b167107958440d10ad93e6e3 WatchSource:0}: Error finding container 0294bd11cbff9a52a082aa1b3fba33fd68829086b167107958440d10ad93e6e3: Status 404 returned error can't find the container with id 0294bd11cbff9a52a082aa1b3fba33fd68829086b167107958440d10ad93e6e3
Nov 02 18:08:22.404147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.404036    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" event={"ID":"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc","Type":"ContainerStarted","Data":"71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595"}
Nov 02 18:08:22.404147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.404086    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" event={"ID":"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc","Type":"ContainerStarted","Data":"0294bd11cbff9a52a082aa1b3fba33fd68829086b167107958440d10ad93e6e3"}
Nov 02 18:08:22.404342 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.404181    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:22.504847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.504823    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"]
Nov 02 18:08:22.506696 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:08:22.506669    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3b0683_9211_4e11_8a35_c2c2a80b1c25.slice/crio-1fe6f0573f87cf1afdd8f20ee5beef30e1f6d58319ac77e2eacaaa2a6020948f WatchSource:0}: Error finding container 1fe6f0573f87cf1afdd8f20ee5beef30e1f6d58319ac77e2eacaaa2a6020948f: Status 404 returned error can't find the container with id 1fe6f0573f87cf1afdd8f20ee5beef30e1f6d58319ac77e2eacaaa2a6020948f
Nov 02 18:08:22.520515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:22.520469    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" podStartSLOduration=1.520457145 podStartE2EDuration="1.520457145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:22.51714433 +0000 UTC m=+2732.774174948" watchObservedRunningTime="2026-11-02 18:08:22.520457145 +0000 UTC m=+2732.777487765"
Nov 02 18:08:23.415017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:23.414968    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" event={"ID":"1e3b0683-9211-4e11-8a35-c2c2a80b1c25","Type":"ContainerStarted","Data":"c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb"}
Nov 02 18:08:23.415507 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:23.415034    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" event={"ID":"1e3b0683-9211-4e11-8a35-c2c2a80b1c25","Type":"ContainerStarted","Data":"1fe6f0573f87cf1afdd8f20ee5beef30e1f6d58319ac77e2eacaaa2a6020948f"}
Nov 02 18:08:23.415507 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:23.415079    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:23.528747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:23.528690    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" podStartSLOduration=2.528676787 podStartE2EDuration="2.528676787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:23.52706629 +0000 UTC m=+2733.784096909" watchObservedRunningTime="2026-11-02 18:08:23.528676787 +0000 UTC m=+2733.785707407"
Nov 02 18:08:29.054089 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:08:29.054047    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:08:31.201500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.201409    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"]
Nov 02 18:08:31.206907 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.206883    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.263641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.263603    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"]
Nov 02 18:08:31.321927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.321446    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2a1bcc\""
Nov 02 18:08:31.322307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.322288    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2a1bcc-dockercfg-s8gc6\""
Nov 02 18:08:31.324297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.324264    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgs7g\" (UniqueName: \"kubernetes.io/projected/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-kube-api-access-jgs7g\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.324823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.324442    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.384384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.384353    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"]
Nov 02 18:08:31.388663 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.388638    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.416337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.416300    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"]
Nov 02 18:08:31.425544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.425513    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgs7g\" (UniqueName: \"kubernetes.io/projected/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-kube-api-access-jgs7g\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.425721 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.425665    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.426544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.426521    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.482564 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.482495    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgs7g\" (UniqueName: \"kubernetes.io/projected/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-kube-api-access-jgs7g\") pod \"payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.521745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.521710    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:31.526545 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.526514    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzkh\" (UniqueName: \"kubernetes.io/projected/e80a322f-cc91-4e81-9607-0698741dec9f-kube-api-access-jfzkh\") pod \"payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.526737 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.526712    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80a322f-cc91-4e81-9607-0698741dec9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.628348 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.628293    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80a322f-cc91-4e81-9607-0698741dec9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.629231 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.628601    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzkh\" (UniqueName: \"kubernetes.io/projected/e80a322f-cc91-4e81-9607-0698741dec9f-kube-api-access-jfzkh\") pod \"payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.630080 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.630001    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80a322f-cc91-4e81-9607-0698741dec9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.686164 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.686135    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzkh\" (UniqueName: \"kubernetes.io/projected/e80a322f-cc91-4e81-9607-0698741dec9f-kube-api-access-jfzkh\") pod \"payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.705034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.705010    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:31.904064 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:31.904031    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"]
Nov 02 18:08:31.905397 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:08:31.905373    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8fb3e7_5cfc_48e5_a16d_31694e71a7e2.slice/crio-45472f0e8a54fee86d2362448599741ce811de3eeb5832bb882c3e6a97fa4459 WatchSource:0}: Error finding container 45472f0e8a54fee86d2362448599741ce811de3eeb5832bb882c3e6a97fa4459: Status 404 returned error can't find the container with id 45472f0e8a54fee86d2362448599741ce811de3eeb5832bb882c3e6a97fa4459
Nov 02 18:08:32.085971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.085944    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"]
Nov 02 18:08:32.086228 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:08:32.086205    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80a322f_cc91_4e81_9607_0698741dec9f.slice/crio-800caf437c0206bd9f65888334ecb1dd111a09eed5c6dcf678231894789364ee WatchSource:0}: Error finding container 800caf437c0206bd9f65888334ecb1dd111a09eed5c6dcf678231894789364ee: Status 404 returned error can't find the container with id 800caf437c0206bd9f65888334ecb1dd111a09eed5c6dcf678231894789364ee
Nov 02 18:08:32.474834 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.474796    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" event={"ID":"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2","Type":"ContainerStarted","Data":"23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841"}
Nov 02 18:08:32.475334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.474860    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" event={"ID":"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2","Type":"ContainerStarted","Data":"45472f0e8a54fee86d2362448599741ce811de3eeb5832bb882c3e6a97fa4459"}
Nov 02 18:08:32.475334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.475038    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:32.476729 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.476705    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" event={"ID":"e80a322f-cc91-4e81-9607-0698741dec9f","Type":"ContainerStarted","Data":"0a3a21bb32196f363b11f606a68b985030a984076b9eef0681a5d6b8dfe3d828"}
Nov 02 18:08:32.476865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.476756    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:32.476865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.476780    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" event={"ID":"e80a322f-cc91-4e81-9607-0698741dec9f","Type":"ContainerStarted","Data":"800caf437c0206bd9f65888334ecb1dd111a09eed5c6dcf678231894789364ee"}
Nov 02 18:08:32.591760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.591708    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" podStartSLOduration=1.591692762 podStartE2EDuration="1.591692762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:32.590849192 +0000 UTC m=+2742.847879822" watchObservedRunningTime="2026-11-02 18:08:32.591692762 +0000 UTC m=+2742.848723383"
Nov 02 18:08:32.710299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:32.710242    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" podStartSLOduration=1.710226638 podStartE2EDuration="1.710226638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:32.706958292 +0000 UTC m=+2742.963988912" watchObservedRunningTime="2026-11-02 18:08:32.710226638 +0000 UTC m=+2742.967257257"
Nov 02 18:08:33.419408 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:33.419377    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:08:34.425179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:34.425149    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:08:36.261787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.261753    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"]
Nov 02 18:08:36.266008 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.265991    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.293346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.293313    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zcvb\" (UniqueName: \"kubernetes.io/projected/146210f3-688e-430f-a109-49704da37205-kube-api-access-6zcvb\") pod \"payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh\" (UID: \"146210f3-688e-430f-a109-49704da37205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.293478 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.293411    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146210f3-688e-430f-a109-49704da37205-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh\" (UID: \"146210f3-688e-430f-a109-49704da37205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.381469 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.381438    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c82472\""
Nov 02 18:08:36.405298 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.381978    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c82472-dockercfg-dgvj6\""
Nov 02 18:08:36.405562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.403256    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146210f3-688e-430f-a109-49704da37205-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh\" (UID: \"146210f3-688e-430f-a109-49704da37205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.405947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.405926    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zcvb\" (UniqueName: \"kubernetes.io/projected/146210f3-688e-430f-a109-49704da37205-kube-api-access-6zcvb\") pod \"payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh\" (UID: \"146210f3-688e-430f-a109-49704da37205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.406350 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.404207    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"]
Nov 02 18:08:36.406733 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.404169    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146210f3-688e-430f-a109-49704da37205-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh\" (UID: \"146210f3-688e-430f-a109-49704da37205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.436069 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.436038    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"]
Nov 02 18:08:36.440573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.440554    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.462969 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.462944    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zcvb\" (UniqueName: \"kubernetes.io/projected/146210f3-688e-430f-a109-49704da37205-kube-api-access-6zcvb\") pod \"payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh\" (UID: \"146210f3-688e-430f-a109-49704da37205\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.486054 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.485961    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"]
Nov 02 18:08:36.507504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.507473    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b8eabf-69c6-46c2-994a-33c73b88601f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.507669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.507513    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjrb\" (UniqueName: \"kubernetes.io/projected/25b8eabf-69c6-46c2-994a-33c73b88601f-kube-api-access-zpjrb\") pod \"payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.581164 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.581072    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:36.608796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.608721    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b8eabf-69c6-46c2-994a-33c73b88601f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.608796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.608758    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjrb\" (UniqueName: \"kubernetes.io/projected/25b8eabf-69c6-46c2-994a-33c73b88601f-kube-api-access-zpjrb\") pod \"payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.609491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.609465    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b8eabf-69c6-46c2-994a-33c73b88601f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.666100 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.666068    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjrb\" (UniqueName: \"kubernetes.io/projected/25b8eabf-69c6-46c2-994a-33c73b88601f-kube-api-access-zpjrb\") pod \"payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.754173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.754108    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:36.959520 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:36.959485    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"]
Nov 02 18:08:36.960530 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:08:36.960502    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146210f3_688e_430f_a109_49704da37205.slice/crio-b7bb67f4518339bb34fb5f5dfd922dd5f1decd46e8c15af91986f6d7aee89e44 WatchSource:0}: Error finding container b7bb67f4518339bb34fb5f5dfd922dd5f1decd46e8c15af91986f6d7aee89e44: Status 404 returned error can't find the container with id b7bb67f4518339bb34fb5f5dfd922dd5f1decd46e8c15af91986f6d7aee89e44
Nov 02 18:08:37.134048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.134024    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"]
Nov 02 18:08:37.135160 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:08:37.135114    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b8eabf_69c6_46c2_994a_33c73b88601f.slice/crio-60308aad79ad48cd385a6523e6cd1c11140f4a1b0766c25a0abfca6e8db9c2c1 WatchSource:0}: Error finding container 60308aad79ad48cd385a6523e6cd1c11140f4a1b0766c25a0abfca6e8db9c2c1: Status 404 returned error can't find the container with id 60308aad79ad48cd385a6523e6cd1c11140f4a1b0766c25a0abfca6e8db9c2c1
Nov 02 18:08:37.521081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.520994    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" event={"ID":"25b8eabf-69c6-46c2-994a-33c73b88601f","Type":"ContainerStarted","Data":"5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1"}
Nov 02 18:08:37.521081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.521051    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" event={"ID":"25b8eabf-69c6-46c2-994a-33c73b88601f","Type":"ContainerStarted","Data":"60308aad79ad48cd385a6523e6cd1c11140f4a1b0766c25a0abfca6e8db9c2c1"}
Nov 02 18:08:37.521599 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.521092    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:37.522826 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.522799    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" event={"ID":"146210f3-688e-430f-a109-49704da37205","Type":"ContainerStarted","Data":"196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545"}
Nov 02 18:08:37.522936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.522846    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" event={"ID":"146210f3-688e-430f-a109-49704da37205","Type":"ContainerStarted","Data":"b7bb67f4518339bb34fb5f5dfd922dd5f1decd46e8c15af91986f6d7aee89e44"}
Nov 02 18:08:37.522936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.522886    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:37.659579 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.659516    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" podStartSLOduration=1.6595007609999999 podStartE2EDuration="1.659500761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:37.633686025 +0000 UTC m=+2747.890716644" watchObservedRunningTime="2026-11-02 18:08:37.659500761 +0000 UTC m=+2747.916531380"
Nov 02 18:08:37.747514 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:37.747461    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" podStartSLOduration=1.747446202 podStartE2EDuration="1.747446202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:37.746609577 +0000 UTC m=+2748.003640198" watchObservedRunningTime="2026-11-02 18:08:37.747446202 +0000 UTC m=+2748.004476821"
Nov 02 18:08:43.486379 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:43.486346    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:08:43.486777 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:43.486423    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:08:48.532334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:48.532307    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:08:48.532733 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:08:48.532372    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:08:59.053787 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:08:59.053746    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:09:29.054484 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:09:29.054445    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:09:30.304468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.304434    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"]
Nov 02 18:09:30.308804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.308784    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.360471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.360439    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-2a1bcc-serving-cert\""
Nov 02 18:09:30.383764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.383722    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.383913 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.383859    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.383962 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.383944    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02d40d-c7bb-4b17-8c06-5570d14496a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.384003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.383980    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbcjv\" (UniqueName: \"kubernetes.io/projected/6c02d40d-c7bb-4b17-8c06-5570d14496a4-kube-api-access-vbcjv\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.413228 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.413187    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"]
Nov 02 18:09:30.485266 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.485234    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbcjv\" (UniqueName: \"kubernetes.io/projected/6c02d40d-c7bb-4b17-8c06-5570d14496a4-kube-api-access-vbcjv\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.485455 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.485368    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.485455 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.485430    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.485584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.485477    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02d40d-c7bb-4b17-8c06-5570d14496a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.485638 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:09:30.485579    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-serving-cert: secret "maas-api-e2e-shared-b-w3-2a1bcc-serving-cert" not found
Nov 02 18:09:30.485694 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:09:30.485642    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls podName:6c02d40d-c7bb-4b17-8c06-5570d14496a4 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:30.985627398 +0000 UTC m=+2801.242657996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls") pod "maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9" (UID: "6c02d40d-c7bb-4b17-8c06-5570d14496a4") : secret "maas-api-e2e-shared-b-w3-2a1bcc-serving-cert" not found
Nov 02 18:09:30.486186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.486156    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02d40d-c7bb-4b17-8c06-5570d14496a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.488238 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.488216    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.542438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.542402    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbcjv\" (UniqueName: \"kubernetes.io/projected/6c02d40d-c7bb-4b17-8c06-5570d14496a4-kube-api-access-vbcjv\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.992068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.992014    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:30.994829 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:30.994793    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:31.222328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:31.222291    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:31.607199 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:31.607173    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"]
Nov 02 18:09:31.609015 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:09:31.608986    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c02d40d_c7bb_4b17_8c06_5570d14496a4.slice/crio-9d084057d34bd2729a18cfd49730aabe00142d9ede2369b5a300641db93dcdac WatchSource:0}: Error finding container 9d084057d34bd2729a18cfd49730aabe00142d9ede2369b5a300641db93dcdac: Status 404 returned error can't find the container with id 9d084057d34bd2729a18cfd49730aabe00142d9ede2369b5a300641db93dcdac
Nov 02 18:09:31.867074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:31.867001    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9" event={"ID":"6c02d40d-c7bb-4b17-8c06-5570d14496a4","Type":"ContainerStarted","Data":"9d084057d34bd2729a18cfd49730aabe00142d9ede2369b5a300641db93dcdac"}
Nov 02 18:09:32.874230 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:32.874136    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9" event={"ID":"6c02d40d-c7bb-4b17-8c06-5570d14496a4","Type":"ContainerStarted","Data":"fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8"}
Nov 02 18:09:32.874230 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:32.874193    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:32.989544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:32.989486    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9" podStartSLOduration=2.082159398 podStartE2EDuration="2.989470995s" totalImagesPullingTime="907.311597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:32.988664191 +0000 UTC m=+2803.245694810" watchObservedRunningTime="2026-11-02 18:09:32.989470995 +0000 UTC m=+2803.246501615"
Nov 02 18:09:38.885976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:09:38.885947    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:09:59.053218 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:09:59.053167    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:10:12.379985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.379951    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"]
Nov 02 18:10:12.385359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.385337    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.435502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.435416    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmgl\" (UniqueName: \"kubernetes.io/projected/14d82bc4-eb24-4c25-b133-468da86a0c3b-kube-api-access-jxmgl\") pod \"payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.435723 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.435578    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d82bc4-eb24-4c25-b133-468da86a0c3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.438933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.438895    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"]
Nov 02 18:10:12.485220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.485188    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-de85ba-dockercfg-t2wg6\""
Nov 02 18:10:12.486255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.486226    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-de85ba\""
Nov 02 18:10:12.528320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.528271    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"]
Nov 02 18:10:12.537549 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.537521    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:12.559072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.559036    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmgl\" (UniqueName: \"kubernetes.io/projected/14d82bc4-eb24-4c25-b133-468da86a0c3b-kube-api-access-jxmgl\") pod \"payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.565067 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.565004    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d82bc4-eb24-4c25-b133-468da86a0c3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.565986 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.565942    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d82bc4-eb24-4c25-b133-468da86a0c3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.593708 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.593675    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"]
Nov 02 18:10:12.617266 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.617234    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmgl\" (UniqueName: \"kubernetes.io/projected/14d82bc4-eb24-4c25-b133-468da86a0c3b-kube-api-access-jxmgl\") pod \"payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.667029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.666906    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9pdn\" (UniqueName: \"kubernetes.io/projected/c2b5740f-4c9d-49be-9bfe-223b63284e66-kube-api-access-m9pdn\") pod \"payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:12.667029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.666956    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5740f-4c9d-49be-9bfe-223b63284e66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:12.699372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.699330    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:12.772147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.770248    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5740f-4c9d-49be-9bfe-223b63284e66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:12.772147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.770568    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9pdn\" (UniqueName: \"kubernetes.io/projected/c2b5740f-4c9d-49be-9bfe-223b63284e66-kube-api-access-m9pdn\") pod \"payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:12.773139 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.773090    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5740f-4c9d-49be-9bfe-223b63284e66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:12.827237 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.827203    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9pdn\" (UniqueName: \"kubernetes.io/projected/c2b5740f-4c9d-49be-9bfe-223b63284e66-kube-api-access-m9pdn\") pod \"payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:12.880793 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:12.880763    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:13.080076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:13.080050    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"]
Nov 02 18:10:13.081057 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:10:13.081030    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d82bc4_eb24_4c25_b133_468da86a0c3b.slice/crio-cf60aedfb9b7720ca9fe496e3e06f14bca36e3c995aba063ec104a12d611ef86 WatchSource:0}: Error finding container cf60aedfb9b7720ca9fe496e3e06f14bca36e3c995aba063ec104a12d611ef86: Status 404 returned error can't find the container with id cf60aedfb9b7720ca9fe496e3e06f14bca36e3c995aba063ec104a12d611ef86
Nov 02 18:10:13.159566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:13.159530    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9" event={"ID":"14d82bc4-eb24-4c25-b133-468da86a0c3b","Type":"ContainerStarted","Data":"dc56408ceded2020826d6d464b543fb32c565b652605c1daace22b799b6f65ea"}
Nov 02 18:10:13.159738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:13.159593    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:13.159738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:13.159609    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9" event={"ID":"14d82bc4-eb24-4c25-b133-468da86a0c3b","Type":"ContainerStarted","Data":"cf60aedfb9b7720ca9fe496e3e06f14bca36e3c995aba063ec104a12d611ef86"}
Nov 02 18:10:13.254990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:13.254966    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"]
Nov 02 18:10:13.256499 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:10:13.256471    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b5740f_4c9d_49be_9bfe_223b63284e66.slice/crio-047e93ea1bab76a369b7509ff1f660d21707c26c7222276bf06d62dd93c91ded WatchSource:0}: Error finding container 047e93ea1bab76a369b7509ff1f660d21707c26c7222276bf06d62dd93c91ded: Status 404 returned error can't find the container with id 047e93ea1bab76a369b7509ff1f660d21707c26c7222276bf06d62dd93c91ded
Nov 02 18:10:13.274793 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:13.274742    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9" podStartSLOduration=1.274728021 podStartE2EDuration="1.274728021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:13.271835662 +0000 UTC m=+2843.528866285" watchObservedRunningTime="2026-11-02 18:10:13.274728021 +0000 UTC m=+2843.531758640"
Nov 02 18:10:14.167911 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:14.167873    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59" event={"ID":"c2b5740f-4c9d-49be-9bfe-223b63284e66","Type":"ContainerStarted","Data":"a8c727461789ce0056f026b2949395c1e6c8f24b9c85fce968e7adc88b1a243a"}
Nov 02 18:10:14.168409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:14.167935    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59" event={"ID":"c2b5740f-4c9d-49be-9bfe-223b63284e66","Type":"ContainerStarted","Data":"047e93ea1bab76a369b7509ff1f660d21707c26c7222276bf06d62dd93c91ded"}
Nov 02 18:10:14.168409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:14.168157    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:14.285097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:14.285046    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59" podStartSLOduration=2.285032616 podStartE2EDuration="2.285032616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:14.280978575 +0000 UTC m=+2844.538009194" watchObservedRunningTime="2026-11-02 18:10:14.285032616 +0000 UTC m=+2844.542063236"
Nov 02 18:10:24.171454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:24.171416    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:10:25.178079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:10:25.178051    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:10:29.053747 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:10:29.053706    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:10:59.052969 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:10:59.052932    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:11:16.954279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.954245    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"]
Nov 02 18:11:16.958947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.958923    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:16.996672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.996638    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-home\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:16.996856 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.996695    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-dshm\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:16.996929 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.996899    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-tmp-dir\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:16.996996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.996947    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6332051-1697-415c-92dd-abfdcd40f827-tls-certs\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:16.997051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.996988    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmp26\" (UniqueName: \"kubernetes.io/projected/f6332051-1697-415c-92dd-abfdcd40f827-kube-api-access-zmp26\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:16.997225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:16.997201    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-model-cache\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.011823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.011792    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-4ecad8\"/\"kube-root-ca.crt\""
Nov 02 18:11:17.016010 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.015972    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"]
Nov 02 18:11:17.040440 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.040408    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-4ecad8\"/\"openshift-service-ca.crt\""
Nov 02 18:11:17.060725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.060694    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-4ecad8\"/\"auth-test-model-4ecad8-kserve-self-signed-certs\""
Nov 02 18:11:17.060917 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.060786    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-4ecad8\"/\"default-dockercfg-ks7m2\""
Nov 02 18:11:17.098690 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.098651    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-dshm\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.098893 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.098747    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-tmp-dir\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.098893 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.098784    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6332051-1697-415c-92dd-abfdcd40f827-tls-certs\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.099028 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.098921    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmp26\" (UniqueName: \"kubernetes.io/projected/f6332051-1697-415c-92dd-abfdcd40f827-kube-api-access-zmp26\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.099093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.099065    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-model-cache\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.099235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.099216    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-tmp-dir\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.099288 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.099261    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-home\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.099402 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.099379    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-model-cache\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.099552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.099534    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-home\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.101162 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.101110    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-dshm\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.101468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.101451    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6332051-1697-415c-92dd-abfdcd40f827-tls-certs\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.157326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.157292    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmp26\" (UniqueName: \"kubernetes.io/projected/f6332051-1697-415c-92dd-abfdcd40f827-kube-api-access-zmp26\") pod \"auth-test-model-4ecad8-kserve-5b74c6745-q8x4b\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") " pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.273290 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.273195    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:17.656663 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:17.656633    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"]
Nov 02 18:11:17.658230 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:11:17.658195    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6332051_1697_415c_92dd_abfdcd40f827.slice/crio-5b1c70a72a53728996f5b8bca3b650fe74fc53850de22ff3b98983aad50fcb1b WatchSource:0}: Error finding container 5b1c70a72a53728996f5b8bca3b650fe74fc53850de22ff3b98983aad50fcb1b: Status 404 returned error can't find the container with id 5b1c70a72a53728996f5b8bca3b650fe74fc53850de22ff3b98983aad50fcb1b
Nov 02 18:11:18.600689 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:18.600648    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b" event={"ID":"f6332051-1697-415c-92dd-abfdcd40f827","Type":"ContainerStarted","Data":"0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678"}
Nov 02 18:11:18.601104 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:18.600701    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b" event={"ID":"f6332051-1697-415c-92dd-abfdcd40f827","Type":"ContainerStarted","Data":"5b1c70a72a53728996f5b8bca3b650fe74fc53850de22ff3b98983aad50fcb1b"}
Nov 02 18:11:18.714670 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:18.714608    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b" podStartSLOduration=2.714594429 podStartE2EDuration="2.714594429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:18.713565068 +0000 UTC m=+2908.970595685" watchObservedRunningTime="2026-11-02 18:11:18.714594429 +0000 UTC m=+2908.971625048"
Nov 02 18:11:19.273881 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:19.273846    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:27.273434 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:27.273395    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:27.287007 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:27.286977    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:27.678252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:27.678173    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:11:29.053471 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:11:29.053438    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:11:56.012723 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:56.012682    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"]
Nov 02 18:11:56.013297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:11:56.012909    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy" containerID="cri-o://7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061" gracePeriod=30
Nov 02 18:11:59.053670 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:11:59.053630    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:12:01.298979 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.298951    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:12:01.360437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360402    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-kube-api-access-z5klk\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-kube-api-access-z5klk\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.360631 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360475    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-podinfo\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.360631 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360507    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istiod-ca-cert\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.360631 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360608    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-data\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.360781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360723    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-envoy\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.360848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360834    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-token\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-token\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.360905 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360888    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-credential-socket\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.360961 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360925    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-certs\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.361017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360960    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-data" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.361084 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.360860    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istiod-ca-cert" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:01.361084 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361063    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-socket\") pod \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\" (UID: \"e8be0887-a6f6-4b19-8272-ec68a0c1b17d\") "
Nov 02 18:12:01.361238 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361209    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-credential-socket" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.361388 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361319    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-certs" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.361687 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361412    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-socket" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.361902 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361877    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.361996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361910    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.361996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361924    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.361996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361936    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.361996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.361949    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.363061 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.363040    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-kube-api-access-z5klk" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "kube-api-access-z5klk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:01.363287 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.363267    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-envoy" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.363458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.363440    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-podinfo" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:01.363553 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.363531    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-token" pod "e8be0887-a6f6-4b19-8272-ec68a0c1b17d" (UID: "e8be0887-a6f6-4b19-8272-ec68a0c1b17d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:01.463051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.463012    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5klk\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-kube-api-access-z5klk\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.463051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.463045    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.463051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.463055    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.463506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.463064    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be0887-a6f6-4b19-8272-ec68a0c1b17d-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:01.894273 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.894237    4956 generic.go:356] "Generic (PLEG): container finished" podID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerID="7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061" exitCode=0
Nov 02 18:12:01.894427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.894301    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"
Nov 02 18:12:01.894427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.894312    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" event={"ID":"e8be0887-a6f6-4b19-8272-ec68a0c1b17d","Type":"ContainerDied","Data":"7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061"}
Nov 02 18:12:01.894427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.894362    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m" event={"ID":"e8be0887-a6f6-4b19-8272-ec68a0c1b17d","Type":"ContainerDied","Data":"19c3bfcab0a401cd3260525784c635bd2aedb39350da3b38bd6cc4543b02a1f8"}
Nov 02 18:12:01.894427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.894390    4956 scope.go:122] "RemoveContainer" containerID="7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061"
Nov 02 18:12:01.904796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.904779    4956 scope.go:122] "RemoveContainer" containerID="7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061"
Nov 02 18:12:01.905050 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:12:01.905029    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061\": container with ID starting with 7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061 not found: ID does not exist" containerID="7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061"
Nov 02 18:12:01.905140 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:01.905059    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061"} err="failed to get container status \"7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061\": rpc error: code = NotFound desc = could not find container \"7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061\": container with ID starting with 7a19a510e19ea55f0e704f224be9854077dd53a68f0d9c5e5278589eb91f1061 not found: ID does not exist"
Nov 02 18:12:02.058372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:02.058333    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"]
Nov 02 18:12:02.107575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:02.107541    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-pjf5m"]
Nov 02 18:12:03.865970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:03.865940    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" path="/var/lib/kubelet/pods/e8be0887-a6f6-4b19-8272-ec68a0c1b17d/volumes"
Nov 02 18:12:09.017800 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.017763    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"]
Nov 02 18:12:09.018229 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.017959    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy" containerID="cri-o://72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d" gracePeriod=30
Nov 02 18:12:09.161662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.161625    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"]
Nov 02 18:12:09.162322 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.162306    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy"
Nov 02 18:12:09.162392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.162324    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy"
Nov 02 18:12:09.162445 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.162435    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8be0887-a6f6-4b19-8272-ec68a0c1b17d" containerName="istio-proxy"
Nov 02 18:12:09.167669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.167646    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.221365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.221276    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"kube-root-ca.crt\""
Nov 02 18:12:09.222385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.221683    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"auth-test-model-413a4e-kserve-self-signed-certs\""
Nov 02 18:12:09.222385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.222290    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"default-dockercfg-qrjpr\""
Nov 02 18:12:09.225795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.225765    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"]
Nov 02 18:12:09.273584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.273503    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-home\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.273758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.273639    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tmp-dir\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.273758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.273676    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tls-certs\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.273841 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.273826    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-model-cache\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.273885 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.273869    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q845\" (UniqueName: \"kubernetes.io/projected/5dd5af35-89b8-4624-bfae-57bbc2aa7095-kube-api-access-5q845\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.273988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.273972    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-dshm\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.324413 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.324382    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"openshift-service-ca.crt\""
Nov 02 18:12:09.375279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375240    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tmp-dir\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375287    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tls-certs\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375343    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-model-cache\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375373    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q845\" (UniqueName: \"kubernetes.io/projected/5dd5af35-89b8-4624-bfae-57bbc2aa7095-kube-api-access-5q845\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375521    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-dshm\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375678    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-home\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375785    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tmp-dir\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375810    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-model-cache\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.375998 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.375980    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-home\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.377939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.377917    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-dshm\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.378237 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.378211    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tls-certs\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.433284 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.433242    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q845\" (UniqueName: \"kubernetes.io/projected/5dd5af35-89b8-4624-bfae-57bbc2aa7095-kube-api-access-5q845\") pod \"auth-test-model-413a4e-kserve-7f566ddf7-fjn6t\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.485043 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.485004    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:09.865733 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:12:09.865702    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dd5af35_89b8_4624_bfae_57bbc2aa7095.slice/crio-ad580d4c5abe8a0913d7642d2595cf33aa993f8c9692879e26d902cb9aeeff1c WatchSource:0}: Error finding container ad580d4c5abe8a0913d7642d2595cf33aa993f8c9692879e26d902cb9aeeff1c: Status 404 returned error can't find the container with id ad580d4c5abe8a0913d7642d2595cf33aa993f8c9692879e26d902cb9aeeff1c
Nov 02 18:12:09.866258 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.866234    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"]
Nov 02 18:12:09.965167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:09.961455    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t" event={"ID":"5dd5af35-89b8-4624-bfae-57bbc2aa7095","Type":"ContainerStarted","Data":"ad580d4c5abe8a0913d7642d2595cf33aa993f8c9692879e26d902cb9aeeff1c"}
Nov 02 18:12:10.968788 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:10.968752    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t" event={"ID":"5dd5af35-89b8-4624-bfae-57bbc2aa7095","Type":"ContainerStarted","Data":"89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558"}
Nov 02 18:12:11.087851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:11.087789    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t" podStartSLOduration=2.087770561 podStartE2EDuration="2.087770561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:11.082383268 +0000 UTC m=+2961.339413890" watchObservedRunningTime="2026-11-02 18:12:11.087770561 +0000 UTC m=+2961.344801182"
Nov 02 18:12:11.485974 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:11.485939    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:14.286689 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.286614    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:12:14.444660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.444627    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-envoy\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.444862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.444751    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-podinfo\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.444862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.444783    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-data\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.444862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.444832    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-token\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-token\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.445029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.444930    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-socket\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445098    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-credential-socket\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445151    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-data" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445211    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-socket" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445216    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-certs\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445325    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istiod-ca-cert\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445345    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-credential-socket" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445390    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-kube-api-access-crmks\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-kube-api-access-crmks\") pod \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\" (UID: \"3db270dc-3200-4d19-9484-c6f0c9e79cd7\") "
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445418    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-certs" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:14.445709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445657    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istiod-ca-cert" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:14.446209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.445990    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.446209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.446014    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.446209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.446029    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.446209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.446044    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.446209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.446061    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.447069 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.447049    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-envoy" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:14.447207 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.447187    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-podinfo" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:14.447631 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.447607    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-token" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:14.447753 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.447735    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-kube-api-access-crmks" pod "3db270dc-3200-4d19-9484-c6f0c9e79cd7" (UID: "3db270dc-3200-4d19-9484-c6f0c9e79cd7"). InnerVolumeSpecName "kube-api-access-crmks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:14.546976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.546889    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmks\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-kube-api-access-crmks\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.546976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.546919    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.546976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.546930    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:14.546976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:14.546939    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db270dc-3200-4d19-9484-c6f0c9e79cd7-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:15.007261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.007225    4956 generic.go:356] "Generic (PLEG): container finished" podID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerID="72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d" exitCode=0
Nov 02 18:12:15.007442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.007280    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"
Nov 02 18:12:15.007442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.007301    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" event={"ID":"3db270dc-3200-4d19-9484-c6f0c9e79cd7","Type":"ContainerDied","Data":"72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d"}
Nov 02 18:12:15.007442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.007356    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6" event={"ID":"3db270dc-3200-4d19-9484-c6f0c9e79cd7","Type":"ContainerDied","Data":"91ad8e6f29aea4352bd129f53a3cb749e82b96e10dc1af12dd6a27a2dc498b96"}
Nov 02 18:12:15.007442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.007379    4956 scope.go:122] "RemoveContainer" containerID="72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d"
Nov 02 18:12:15.023089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.023063    4956 scope.go:122] "RemoveContainer" containerID="72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d"
Nov 02 18:12:15.023498 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:12:15.023469    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d\": container with ID starting with 72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d not found: ID does not exist" containerID="72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d"
Nov 02 18:12:15.023655 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.023508    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d"} err="failed to get container status \"72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d\": rpc error: code = NotFound desc = could not find container \"72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d\": container with ID starting with 72d09fc209063d136a5c04accefd0126f86063fa2bb729e3c0b0948ae7ede66d not found: ID does not exist"
Nov 02 18:12:15.220714 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.220676    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"]
Nov 02 18:12:15.240193 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.240164    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-8rpv6"]
Nov 02 18:12:15.866445 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:15.866412    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" path="/var/lib/kubelet/pods/3db270dc-3200-4d19-9484-c6f0c9e79cd7/volumes"
Nov 02 18:12:19.486202 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:19.486162    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:19.498844 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:19.498817    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:20.050080 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:20.050052    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:12:29.054462 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:12:29.054424    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:12:44.619700 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.619662    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"]
Nov 02 18:12:44.622106 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.620272    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy"
Nov 02 18:12:44.622106 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.620285    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy"
Nov 02 18:12:44.622106 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.620416    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db270dc-3200-4d19-9484-c6f0c9e79cd7" containerName="istio-proxy"
Nov 02 18:12:44.623745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.623730    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.635600 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.635572    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-home\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.635724 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.635614    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vflb\" (UniqueName: \"kubernetes.io/projected/8b83246f-76b0-477c-9eff-2f7800508bbd-kube-api-access-7vflb\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.635724 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.635637    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-tmp-dir\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.635811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.635735    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-dshm\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.635811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.635756    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-model-cache\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.635811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.635772    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b83246f-76b0-477c-9eff-2f7800508bbd-tls-certs\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.689714 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.689686    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"openshift-service-ca.crt\""
Nov 02 18:12:44.725398 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.725364    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"kube-root-ca.crt\""
Nov 02 18:12:44.725686 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.725539    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"default-dockercfg-cc5b7\""
Nov 02 18:12:44.725686 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.725588    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"auth-test-model-2a1bcc-kserve-self-signed-certs\""
Nov 02 18:12:44.729045 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.729021    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"]
Nov 02 18:12:44.737073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737048    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-home\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737231 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737107    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vflb\" (UniqueName: \"kubernetes.io/projected/8b83246f-76b0-477c-9eff-2f7800508bbd-kube-api-access-7vflb\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737231 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737157    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-tmp-dir\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737231 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737199    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-dshm\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737231 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737222    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-model-cache\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737448 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737244    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b83246f-76b0-477c-9eff-2f7800508bbd-tls-certs\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737509 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737494    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-home\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737567 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737550    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-tmp-dir\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.737698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.737666    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-model-cache\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.739734 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.739712    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-dshm\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.740003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.739988    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b83246f-76b0-477c-9eff-2f7800508bbd-tls-certs\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.793627 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.793594    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vflb\" (UniqueName: \"kubernetes.io/projected/8b83246f-76b0-477c-9eff-2f7800508bbd-kube-api-access-7vflb\") pod \"auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:44.936440 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:44.936354    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:45.319760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:45.315882    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"]
Nov 02 18:12:46.239395 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:46.239354    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr" event={"ID":"8b83246f-76b0-477c-9eff-2f7800508bbd","Type":"ContainerStarted","Data":"f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735"}
Nov 02 18:12:46.239864 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:46.239403    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr" event={"ID":"8b83246f-76b0-477c-9eff-2f7800508bbd","Type":"ContainerStarted","Data":"ff0a22764b10cd42ba57a47cd0c6ac36e78417a27e4bdc72389cf17943058796"}
Nov 02 18:12:46.354842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:46.354780    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr" podStartSLOduration=2.3547462230000002 podStartE2EDuration="2.354746223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:46.352174169 +0000 UTC m=+2996.609204791" watchObservedRunningTime="2026-11-02 18:12:46.354746223 +0000 UTC m=+2996.611776880"
Nov 02 18:12:46.937405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:46.937370    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:50.266649 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:50.266620    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:12:50.268473 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:50.268450    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:12:54.937091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:54.937049    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:54.950639 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:54.950610    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:55.157882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:55.157850    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"]
Nov 02 18:12:55.363871 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:12:55.363841    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:12:59.053182 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:12:59.053149    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:13:00.158965 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.158873    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb" podUID="ec782aa8-8d0f-4822-840a-da1d25dfae8f" containerName="payload-processing" containerID="cri-o://4f8f5f35e24f86166d7e1e2907481c0fd04fe61b149b5bb15eea2d79d303cbaf" gracePeriod=25
Nov 02 18:13:00.390283 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.390252    4956 generic.go:356] "Generic (PLEG): container finished" podID="ec782aa8-8d0f-4822-840a-da1d25dfae8f" containerID="4f8f5f35e24f86166d7e1e2907481c0fd04fe61b149b5bb15eea2d79d303cbaf" exitCode=0
Nov 02 18:13:00.390509 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.390343    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb" event={"ID":"ec782aa8-8d0f-4822-840a-da1d25dfae8f","Type":"ContainerDied","Data":"4f8f5f35e24f86166d7e1e2907481c0fd04fe61b149b5bb15eea2d79d303cbaf"}
Nov 02 18:13:00.438678 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.438650    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:13:00.547641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.547611    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec782aa8-8d0f-4822-840a-da1d25dfae8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec782aa8-8d0f-4822-840a-da1d25dfae8f-plugins-config-volume\") pod \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") "
Nov 02 18:13:00.547813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.547705    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec782aa8-8d0f-4822-840a-da1d25dfae8f-kube-api-access-c65c9\" (UniqueName: \"kubernetes.io/projected/ec782aa8-8d0f-4822-840a-da1d25dfae8f-kube-api-access-c65c9\") pod \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\" (UID: \"ec782aa8-8d0f-4822-840a-da1d25dfae8f\") "
Nov 02 18:13:00.547981 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.547958    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec782aa8-8d0f-4822-840a-da1d25dfae8f-plugins-config-volume" pod "ec782aa8-8d0f-4822-840a-da1d25dfae8f" (UID: "ec782aa8-8d0f-4822-840a-da1d25dfae8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:00.549869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.549844    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec782aa8-8d0f-4822-840a-da1d25dfae8f-kube-api-access-c65c9" pod "ec782aa8-8d0f-4822-840a-da1d25dfae8f" (UID: "ec782aa8-8d0f-4822-840a-da1d25dfae8f"). InnerVolumeSpecName "kube-api-access-c65c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:00.649048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.649017    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec782aa8-8d0f-4822-840a-da1d25dfae8f-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:00.649048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:00.649047    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65c9\" (UniqueName: \"kubernetes.io/projected/ec782aa8-8d0f-4822-840a-da1d25dfae8f-kube-api-access-c65c9\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:01.397864 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:01.397834    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"
Nov 02 18:13:01.398419 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:01.397872    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb" event={"ID":"ec782aa8-8d0f-4822-840a-da1d25dfae8f","Type":"ContainerDied","Data":"7e464ecce75727fb93be34119e7cbcffe3d08fcc26cc6b6a71f087398f65c618"}
Nov 02 18:13:01.398419 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:01.397937    4956 scope.go:122] "RemoveContainer" containerID="4f8f5f35e24f86166d7e1e2907481c0fd04fe61b149b5bb15eea2d79d303cbaf"
Nov 02 18:13:01.561435 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:01.561401    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"]
Nov 02 18:13:01.610049 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:01.610015    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb1633-5c4798c695-26jgb"]
Nov 02 18:13:01.871726 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:01.871687    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec782aa8-8d0f-4822-840a-da1d25dfae8f" path="/var/lib/kubelet/pods/ec782aa8-8d0f-4822-840a-da1d25dfae8f/volumes"
Nov 02 18:13:29.056319 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:13:29.056275    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:13:36.608295 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:36.608260    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"]
Nov 02 18:13:36.608747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:36.608499    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy" containerID="cri-o://08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb" gracePeriod=30
Nov 02 18:13:41.889787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.889707    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 18:13:41.959897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.959857    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-certs\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960047 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.959984    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-token\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-token\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960047 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960020    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-envoy\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960247 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960051    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-kube-api-access-k4klb\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-kube-api-access-k4klb\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960247 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960085    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-data\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960247 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960106    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ac57e2-619f-4ada-abfb-2369d107221c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac57e2-619f-4ada-abfb-2369d107221c-istiod-ca-cert\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960247 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960166    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-certs" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:41.960454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960280    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-podinfo\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960329    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-credential-socket\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960418    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-socket\") pod \"a1ac57e2-619f-4ada-abfb-2369d107221c\" (UID: \"a1ac57e2-619f-4ada-abfb-2369d107221c\") "
Nov 02 18:13:41.960748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960416    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-data" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960594    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-credential-socket" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.960747    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-socket" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.961039    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ac57e2-619f-4ada-abfb-2369d107221c-istiod-ca-cert" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.961370    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.961399    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac57e2-619f-4ada-abfb-2369d107221c-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.961416    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.961429    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:41.961547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.961444    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:41.962695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.962670    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-token" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:41.963074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.963040    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-podinfo" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:13:41.963182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.963085    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-kube-api-access-k4klb" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "kube-api-access-k4klb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:41.963249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:41.963187    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-envoy" pod "a1ac57e2-619f-4ada-abfb-2369d107221c" (UID: "a1ac57e2-619f-4ada-abfb-2369d107221c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:42.062764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.062731    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:42.062764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.062761    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:42.062764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.062771    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4klb\" (UniqueName: \"kubernetes.io/projected/a1ac57e2-619f-4ada-abfb-2369d107221c-kube-api-access-k4klb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:42.063002 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.062781    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ac57e2-619f-4ada-abfb-2369d107221c-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:42.666591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.666559    4956 generic.go:356] "Generic (PLEG): container finished" podID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerID="08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb" exitCode=0
Nov 02 18:13:42.666794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.666654    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"
Nov 02 18:13:42.666794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.666653    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" event={"ID":"a1ac57e2-619f-4ada-abfb-2369d107221c","Type":"ContainerDied","Data":"08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb"}
Nov 02 18:13:42.666889 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.666793    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9" event={"ID":"a1ac57e2-619f-4ada-abfb-2369d107221c","Type":"ContainerDied","Data":"b783fb9623bd5ff82bee792da74fb3ac89e8f823135874280d81966faad4b9e4"}
Nov 02 18:13:42.666889 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.666825    4956 scope.go:122] "RemoveContainer" containerID="08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb"
Nov 02 18:13:42.679927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.679904    4956 scope.go:122] "RemoveContainer" containerID="08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb"
Nov 02 18:13:42.680256 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:13:42.680236    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb\": container with ID starting with 08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb not found: ID does not exist" containerID="08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb"
Nov 02 18:13:42.680335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.680266    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb"} err="failed to get container status \"08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb\": rpc error: code = NotFound desc = could not find container \"08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb\": container with ID starting with 08042640b2a43f0e65197f221fa9b52669fb849af433b30633bd6728b2f325eb not found: ID does not exist"
Nov 02 18:13:42.879900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.879863    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"]
Nov 02 18:13:42.929775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:42.929690    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-x2fx9"]
Nov 02 18:13:43.865258 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:43.865228    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" path="/var/lib/kubelet/pods/a1ac57e2-619f-4ada-abfb-2369d107221c/volumes"
Nov 02 18:13:51.511917 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:51.511875    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"]
Nov 02 18:13:51.566703 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:51.566664    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"]
Nov 02 18:13:51.757244 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:51.757209    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"]
Nov 02 18:13:51.757430 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:51.757407    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy" containerID="cri-o://36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a" gracePeriod=30
Nov 02 18:13:56.512323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.512262    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59" podUID="c2b5740f-4c9d-49be-9bfe-223b63284e66" containerName="payload-processing" containerID="cri-o://a8c727461789ce0056f026b2949395c1e6c8f24b9c85fce968e7adc88b1a243a" gracePeriod=25
Nov 02 18:13:56.567918 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.567876    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9" podUID="14d82bc4-eb24-4c25-b133-468da86a0c3b" containerName="payload-pre-processing" containerID="cri-o://dc56408ceded2020826d6d464b543fb32c565b652605c1daace22b799b6f65ea" gracePeriod=25
Nov 02 18:13:56.791434 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.791178    4956 generic.go:356] "Generic (PLEG): container finished" podID="c2b5740f-4c9d-49be-9bfe-223b63284e66" containerID="a8c727461789ce0056f026b2949395c1e6c8f24b9c85fce968e7adc88b1a243a" exitCode=0
Nov 02 18:13:56.791434 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.791274    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59" event={"ID":"c2b5740f-4c9d-49be-9bfe-223b63284e66","Type":"ContainerDied","Data":"a8c727461789ce0056f026b2949395c1e6c8f24b9c85fce968e7adc88b1a243a"}
Nov 02 18:13:56.798673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.798579    4956 generic.go:356] "Generic (PLEG): container finished" podID="14d82bc4-eb24-4c25-b133-468da86a0c3b" containerID="dc56408ceded2020826d6d464b543fb32c565b652605c1daace22b799b6f65ea" exitCode=0
Nov 02 18:13:56.798673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.798653    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9" event={"ID":"14d82bc4-eb24-4c25-b133-468da86a0c3b","Type":"ContainerDied","Data":"dc56408ceded2020826d6d464b543fb32c565b652605c1daace22b799b6f65ea"}
Nov 02 18:13:56.900866 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.900828    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:13:56.904850 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:56.904824    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:13:57.017585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.017556    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:13:57.060557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.060464    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d82bc4-eb24-4c25-b133-468da86a0c3b-kube-api-access-jxmgl\" (UniqueName: \"kubernetes.io/projected/14d82bc4-eb24-4c25-b133-468da86a0c3b-kube-api-access-jxmgl\") pod \"14d82bc4-eb24-4c25-b133-468da86a0c3b\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") "
Nov 02 18:13:57.060732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.060587    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d82bc4-eb24-4c25-b133-468da86a0c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d82bc4-eb24-4c25-b133-468da86a0c3b-plugins-config-volume\") pod \"14d82bc4-eb24-4c25-b133-468da86a0c3b\" (UID: \"14d82bc4-eb24-4c25-b133-468da86a0c3b\") "
Nov 02 18:13:57.060732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.060615    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b5740f-4c9d-49be-9bfe-223b63284e66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5740f-4c9d-49be-9bfe-223b63284e66-plugins-config-volume\") pod \"c2b5740f-4c9d-49be-9bfe-223b63284e66\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") "
Nov 02 18:13:57.060732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.060713    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b5740f-4c9d-49be-9bfe-223b63284e66-kube-api-access-m9pdn\" (UniqueName: \"kubernetes.io/projected/c2b5740f-4c9d-49be-9bfe-223b63284e66-kube-api-access-m9pdn\") pod \"c2b5740f-4c9d-49be-9bfe-223b63284e66\" (UID: \"c2b5740f-4c9d-49be-9bfe-223b63284e66\") "
Nov 02 18:13:57.061056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.061027    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d82bc4-eb24-4c25-b133-468da86a0c3b-plugins-config-volume" pod "14d82bc4-eb24-4c25-b133-468da86a0c3b" (UID: "14d82bc4-eb24-4c25-b133-468da86a0c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:57.061056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.061027    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b5740f-4c9d-49be-9bfe-223b63284e66-plugins-config-volume" pod "c2b5740f-4c9d-49be-9bfe-223b63284e66" (UID: "c2b5740f-4c9d-49be-9bfe-223b63284e66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:57.061269 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.061247    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d82bc4-eb24-4c25-b133-468da86a0c3b-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.061269 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.061264    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b5740f-4c9d-49be-9bfe-223b63284e66-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.062783 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.062756    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d82bc4-eb24-4c25-b133-468da86a0c3b-kube-api-access-jxmgl" pod "14d82bc4-eb24-4c25-b133-468da86a0c3b" (UID: "14d82bc4-eb24-4c25-b133-468da86a0c3b"). InnerVolumeSpecName "kube-api-access-jxmgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:57.062870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.062855    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b5740f-4c9d-49be-9bfe-223b63284e66-kube-api-access-m9pdn" pod "c2b5740f-4c9d-49be-9bfe-223b63284e66" (UID: "c2b5740f-4c9d-49be-9bfe-223b63284e66"). InnerVolumeSpecName "kube-api-access-m9pdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:57.162555 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162511    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-data\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.162747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162569    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-credential-socket\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.162747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162655    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istiod-ca-cert\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.162747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162707    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-envoy\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.162926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162770    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-podinfo\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.162926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162806    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-token\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-token\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.162926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162836    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-data" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:57.162926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162853    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-certs\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.163262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.162956    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-kube-api-access-t84rl\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-kube-api-access-t84rl\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.163262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163016    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-socket\") pod \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\" (UID: \"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4\") "
Nov 02 18:13:57.163262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163208    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-certs" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:57.163440 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163413    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-credential-socket" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:57.163492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163425    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-socket" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:57.163678 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163639    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istiod-ca-cert" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:57.163769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163706    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.163769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163727    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9pdn\" (UniqueName: \"kubernetes.io/projected/c2b5740f-4c9d-49be-9bfe-223b63284e66-kube-api-access-m9pdn\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.163769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163744    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.163769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163759    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.164181 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163774    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.164181 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.163789    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmgl\" (UniqueName: \"kubernetes.io/projected/14d82bc4-eb24-4c25-b133-468da86a0c3b-kube-api-access-jxmgl\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.165707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.165672    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-token" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:57.166333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.166296    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-kube-api-access-t84rl" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "kube-api-access-t84rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:57.166695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.166674    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-podinfo" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:13:57.166695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.166682    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-envoy" pod "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" (UID: "b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:57.264688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.264653    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.264688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.264684    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.264930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.264700    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84rl\" (UniqueName: \"kubernetes.io/projected/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-kube-api-access-t84rl\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.264930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.264714    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.264930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.264728    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:57.806601 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.806518    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"
Nov 02 18:13:57.806601 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.806563    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59" event={"ID":"c2b5740f-4c9d-49be-9bfe-223b63284e66","Type":"ContainerDied","Data":"047e93ea1bab76a369b7509ff1f660d21707c26c7222276bf06d62dd93c91ded"}
Nov 02 18:13:57.807103 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.806631    4956 scope.go:122] "RemoveContainer" containerID="a8c727461789ce0056f026b2949395c1e6c8f24b9c85fce968e7adc88b1a243a"
Nov 02 18:13:57.808217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.808193    4956 generic.go:356] "Generic (PLEG): container finished" podID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerID="36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a" exitCode=0
Nov 02 18:13:57.808310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.808225    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" event={"ID":"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4","Type":"ContainerDied","Data":"36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a"}
Nov 02 18:13:57.808310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.808255    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"
Nov 02 18:13:57.808310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.808286    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw" event={"ID":"b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4","Type":"ContainerDied","Data":"7ec940327baaebc8ddbfdcf6806887d1a59ad3aa83a0dd64bed52d9f245fdc3d"}
Nov 02 18:13:57.814733 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.814688    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"
Nov 02 18:13:57.815321 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.815287    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9" event={"ID":"14d82bc4-eb24-4c25-b133-468da86a0c3b","Type":"ContainerDied","Data":"cf60aedfb9b7720ca9fe496e3e06f14bca36e3c995aba063ec104a12d611ef86"}
Nov 02 18:13:57.824355 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.824317    4956 scope.go:122] "RemoveContainer" containerID="36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a"
Nov 02 18:13:57.841346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.841325    4956 scope.go:122] "RemoveContainer" containerID="36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a"
Nov 02 18:13:57.841654 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:13:57.841633    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a\": container with ID starting with 36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a not found: ID does not exist" containerID="36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a"
Nov 02 18:13:57.841719 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.841663    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a"} err="failed to get container status \"36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a\": rpc error: code = NotFound desc = could not find container \"36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a\": container with ID starting with 36714a6d450379af510240afb8d1af074e2e4cd5d09035b0910c665e0b392b1a not found: ID does not exist"
Nov 02 18:13:57.841719 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.841685    4956 scope.go:122] "RemoveContainer" containerID="dc56408ceded2020826d6d464b543fb32c565b652605c1daace22b799b6f65ea"
Nov 02 18:13:57.973308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:57.973274    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"]
Nov 02 18:13:58.022836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:58.022807    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-8wmdw"]
Nov 02 18:13:58.083776 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:58.083689    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"]
Nov 02 18:13:58.089802 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:58.089774    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-de85ba-768659dd79-25lp9"]
Nov 02 18:13:58.197376 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:58.197343    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"]
Nov 02 18:13:58.200839 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:58.200815    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-de85ba-584767b595-2vq59"]
Nov 02 18:13:59.053408 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:13:59.053371    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:13:59.866166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:59.866112    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d82bc4-eb24-4c25-b133-468da86a0c3b" path="/var/lib/kubelet/pods/14d82bc4-eb24-4c25-b133-468da86a0c3b/volumes"
Nov 02 18:13:59.866592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:59.866578    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" path="/var/lib/kubelet/pods/b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4/volumes"
Nov 02 18:13:59.866927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:13:59.866915    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b5740f-4c9d-49be-9bfe-223b63284e66" path="/var/lib/kubelet/pods/c2b5740f-4c9d-49be-9bfe-223b63284e66/volumes"
Nov 02 18:14:15.402925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:15.402883    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"]
Nov 02 18:14:29.053106 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:14:29.053072    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:14:30.424370 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.424286    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t" podUID="5dd5af35-89b8-4624-bfae-57bbc2aa7095" containerName="main" containerID="cri-o://89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558" gracePeriod=45
Nov 02 18:14:30.686059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.685991    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:14:30.782071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782041    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tls-certs\") pod \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") "
Nov 02 18:14:30.782308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782098    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-model-cache\") pod \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") "
Nov 02 18:14:30.782308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782202    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd5af35-89b8-4624-bfae-57bbc2aa7095-kube-api-access-5q845\" (UniqueName: \"kubernetes.io/projected/5dd5af35-89b8-4624-bfae-57bbc2aa7095-kube-api-access-5q845\") pod \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") "
Nov 02 18:14:30.782308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782271    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-home\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-home\") pod \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") "
Nov 02 18:14:30.782502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782322    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-dshm\") pod \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") "
Nov 02 18:14:30.782502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782348    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tmp-dir\") pod \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\" (UID: \"5dd5af35-89b8-4624-bfae-57bbc2aa7095\") "
Nov 02 18:14:30.782502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782438    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-model-cache" pod "5dd5af35-89b8-4624-bfae-57bbc2aa7095" (UID: "5dd5af35-89b8-4624-bfae-57bbc2aa7095"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:30.782663 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782591    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-home" pod "5dd5af35-89b8-4624-bfae-57bbc2aa7095" (UID: "5dd5af35-89b8-4624-bfae-57bbc2aa7095"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:30.782663 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782627    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tmp-dir" pod "5dd5af35-89b8-4624-bfae-57bbc2aa7095" (UID: "5dd5af35-89b8-4624-bfae-57bbc2aa7095"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:30.782898 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782852    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:30.782898 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782877    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:30.782898 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.782892    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:30.784576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.784554    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-dshm" pod "5dd5af35-89b8-4624-bfae-57bbc2aa7095" (UID: "5dd5af35-89b8-4624-bfae-57bbc2aa7095"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:30.784685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.784630    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tls-certs" pod "5dd5af35-89b8-4624-bfae-57bbc2aa7095" (UID: "5dd5af35-89b8-4624-bfae-57bbc2aa7095"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:14:30.784746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.784705    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd5af35-89b8-4624-bfae-57bbc2aa7095-kube-api-access-5q845" pod "5dd5af35-89b8-4624-bfae-57bbc2aa7095" (UID: "5dd5af35-89b8-4624-bfae-57bbc2aa7095"). InnerVolumeSpecName "kube-api-access-5q845". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:30.884657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.884618    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q845\" (UniqueName: \"kubernetes.io/projected/5dd5af35-89b8-4624-bfae-57bbc2aa7095-kube-api-access-5q845\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:30.884657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.884651    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd5af35-89b8-4624-bfae-57bbc2aa7095-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:30.884657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:30.884660    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd5af35-89b8-4624-bfae-57bbc2aa7095-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:31.029175 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.029114    4956 generic.go:356] "Generic (PLEG): container finished" podID="5dd5af35-89b8-4624-bfae-57bbc2aa7095" containerID="89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558" exitCode=0
Nov 02 18:14:31.029368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.029179    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"
Nov 02 18:14:31.029368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.029212    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t" event={"ID":"5dd5af35-89b8-4624-bfae-57bbc2aa7095","Type":"ContainerDied","Data":"89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558"}
Nov 02 18:14:31.029368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.029259    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t" event={"ID":"5dd5af35-89b8-4624-bfae-57bbc2aa7095","Type":"ContainerDied","Data":"ad580d4c5abe8a0913d7642d2595cf33aa993f8c9692879e26d902cb9aeeff1c"}
Nov 02 18:14:31.029368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.029280    4956 scope.go:122] "RemoveContainer" containerID="89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558"
Nov 02 18:14:31.039636 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.039616    4956 scope.go:122] "RemoveContainer" containerID="89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558"
Nov 02 18:14:31.039938 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:14:31.039917    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558\": container with ID starting with 89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558 not found: ID does not exist" containerID="89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558"
Nov 02 18:14:31.039990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.039948    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558"} err="failed to get container status \"89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558\": rpc error: code = NotFound desc = could not find container \"89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558\": container with ID starting with 89c5e9a65ca930724b16a20759b864fdd5a631a4f7372a7ec72cd6e58b04c558 not found: ID does not exist"
Nov 02 18:14:31.193622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.193575    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"]
Nov 02 18:14:31.243066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.243028    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/auth-test-model-413a4e-kserve-7f566ddf7-fjn6t"]
Nov 02 18:14:31.867491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:31.867451    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd5af35-89b8-4624-bfae-57bbc2aa7095" path="/var/lib/kubelet/pods/5dd5af35-89b8-4624-bfae-57bbc2aa7095/volumes"
Nov 02 18:14:55.823573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.823532    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"]
Nov 02 18:14:55.825079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825047    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy"
Nov 02 18:14:55.825079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825079    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825142    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ec782aa8-8d0f-4822-840a-da1d25dfae8f" containerName="payload-processing"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825150    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec782aa8-8d0f-4822-840a-da1d25dfae8f" containerName="payload-processing"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825165    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825173    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825219    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c2b5740f-4c9d-49be-9bfe-223b63284e66" containerName="payload-processing"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825227    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b5740f-4c9d-49be-9bfe-223b63284e66" containerName="payload-processing"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825240    4956 cpu_manager.go:414] "containerMap: removing container" podUID="14d82bc4-eb24-4c25-b133-468da86a0c3b" containerName="payload-pre-processing"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825248    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d82bc4-eb24-4c25-b133-468da86a0c3b" containerName="payload-pre-processing"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825256    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5dd5af35-89b8-4624-bfae-57bbc2aa7095" containerName="main"
Nov 02 18:14:55.825331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825262    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd5af35-89b8-4624-bfae-57bbc2aa7095" containerName="main"
Nov 02 18:14:55.825817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825417    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b5740f-4c9d-49be-9bfe-223b63284e66" containerName="payload-processing"
Nov 02 18:14:55.825817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825432    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d82bc4-eb24-4c25-b133-468da86a0c3b" containerName="payload-pre-processing"
Nov 02 18:14:55.825817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825442    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd5af35-89b8-4624-bfae-57bbc2aa7095" containerName="main"
Nov 02 18:14:55.825817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825458    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b912ddaa-0dc7-4ebd-9afb-f188eae3e1a4" containerName="istio-proxy"
Nov 02 18:14:55.825817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825471    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec782aa8-8d0f-4822-840a-da1d25dfae8f" containerName="payload-processing"
Nov 02 18:14:55.825817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.825481    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ac57e2-619f-4ada-abfb-2369d107221c" containerName="istio-proxy"
Nov 02 18:14:55.829614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.829588    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:55.880597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.880563    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"]
Nov 02 18:14:55.891220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.891174    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2pp7\" (UniqueName: \"kubernetes.io/projected/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-kube-api-access-w2pp7\") pod \"payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") " pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:55.891446 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.891423    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-plugins-config-volume\") pod \"payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") " pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:55.926637 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.926612    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-eb709f\""
Nov 02 18:14:55.926800 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.926710    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-eb709f-dockercfg-w48tq\""
Nov 02 18:14:55.992673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.992638    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2pp7\" (UniqueName: \"kubernetes.io/projected/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-kube-api-access-w2pp7\") pod \"payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") " pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:55.992874 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.992756    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-plugins-config-volume\") pod \"payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") " pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:55.993492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:55.993472    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-plugins-config-volume\") pod \"payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") " pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:56.052958 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:56.052924    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2pp7\" (UniqueName: \"kubernetes.io/projected/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-kube-api-access-w2pp7\") pod \"payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") " pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:56.146623 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:56.146543    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:56.521976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:56.521944    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"]
Nov 02 18:14:56.523536 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:14:56.523502    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecdd73b_975c_4145_8da6_7f9edd9d8e61.slice/crio-345ae1db70e5411e46161dc065abd95a2ffe3f4ce69d5d58e1c73b2eb5932e6f WatchSource:0}: Error finding container 345ae1db70e5411e46161dc065abd95a2ffe3f4ce69d5d58e1c73b2eb5932e6f: Status 404 returned error can't find the container with id 345ae1db70e5411e46161dc065abd95a2ffe3f4ce69d5d58e1c73b2eb5932e6f
Nov 02 18:14:57.195693 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:57.195653    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866" event={"ID":"5ecdd73b-975c-4145-8da6-7f9edd9d8e61","Type":"ContainerStarted","Data":"7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb"}
Nov 02 18:14:57.196096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:57.195711    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866" event={"ID":"5ecdd73b-975c-4145-8da6-7f9edd9d8e61","Type":"ContainerStarted","Data":"345ae1db70e5411e46161dc065abd95a2ffe3f4ce69d5d58e1c73b2eb5932e6f"}
Nov 02 18:14:57.196096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:57.195738    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:14:57.311014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:14:57.310966    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866" podStartSLOduration=2.31095213 podStartE2EDuration="2.31095213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:57.307608753 +0000 UTC m=+3127.564639374" watchObservedRunningTime="2026-11-02 18:14:57.31095213 +0000 UTC m=+3127.567982750"
Nov 02 18:14:59.052814 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:14:59.052776    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:15:00.172310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.172273    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"]
Nov 02 18:15:00.176549 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.176530    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.197595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.197567    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"]
Nov 02 18:15:00.201526 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.201505    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.215563 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.215538    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"]
Nov 02 18:15:00.219665 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.219632    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.222083 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.222050    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"]
Nov 02 18:15:00.226172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.226151    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.244442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.244415    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:15:00.244811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.244793    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c4b46d-ff39-476f-8582-06ddbf12afb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.244856 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.244823    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmwhz\" (UniqueName: \"kubernetes.io/projected/d2c4b46d-ff39-476f-8582-06ddbf12afb5-kube-api-access-dmwhz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.244926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.244907    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2lw\" (UniqueName: \"kubernetes.io/projected/fef0830a-21d5-4002-9bfe-473d9bcc40c5-kube-api-access-xj2lw\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.245012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.244998    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef0830a-21d5-4002-9bfe-473d9bcc40c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.278377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.278329    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"]
Nov 02 18:15:00.325480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.325421    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"]
Nov 02 18:15:00.346971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.346933    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gwzx\" (UniqueName: \"kubernetes.io/projected/430b80c8-354f-4332-b6c7-86c604facaaf-kube-api-access-7gwzx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.347194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347073    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef0830a-21d5-4002-9bfe-473d9bcc40c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.347194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347158    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.347341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347197    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2sgw\" (UniqueName: \"kubernetes.io/projected/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-kube-api-access-p2sgw\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.347433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347412    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c4b46d-ff39-476f-8582-06ddbf12afb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.347508 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347452    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmwhz\" (UniqueName: \"kubernetes.io/projected/d2c4b46d-ff39-476f-8582-06ddbf12afb5-kube-api-access-dmwhz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.347566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347537    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430b80c8-354f-4332-b6c7-86c604facaaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.347621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347570    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2lw\" (UniqueName: \"kubernetes.io/projected/fef0830a-21d5-4002-9bfe-473d9bcc40c5-kube-api-access-xj2lw\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.347882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.347861    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef0830a-21d5-4002-9bfe-473d9bcc40c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.348242 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.348216    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c4b46d-ff39-476f-8582-06ddbf12afb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.375424 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.375191    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"]
Nov 02 18:15:00.392899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.392862    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"]
Nov 02 18:15:00.408314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.408283    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmwhz\" (UniqueName: \"kubernetes.io/projected/d2c4b46d-ff39-476f-8582-06ddbf12afb5-kube-api-access-dmwhz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.412385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.412352    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2lw\" (UniqueName: \"kubernetes.io/projected/fef0830a-21d5-4002-9bfe-473d9bcc40c5-kube-api-access-xj2lw\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.448748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.448657    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430b80c8-354f-4332-b6c7-86c604facaaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.448748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.448708    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gwzx\" (UniqueName: \"kubernetes.io/projected/430b80c8-354f-4332-b6c7-86c604facaaf-kube-api-access-7gwzx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.448984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.448773    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.448984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.448796    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sgw\" (UniqueName: \"kubernetes.io/projected/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-kube-api-access-p2sgw\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.449459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.449433    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430b80c8-354f-4332-b6c7-86c604facaaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.449459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.449444    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.490018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.489986    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:00.504985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.504959    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gwzx\" (UniqueName: \"kubernetes.io/projected/430b80c8-354f-4332-b6c7-86c604facaaf-kube-api-access-7gwzx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.505114 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.505026    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2sgw\" (UniqueName: \"kubernetes.io/projected/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-kube-api-access-p2sgw\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.517083 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.516817    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:00.537449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.537274    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:00.545536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.545494    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:00.884391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.884363    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"]
Nov 02 18:15:00.885019 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:15:00.884998    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c4b46d_ff39_476f_8582_06ddbf12afb5.slice/crio-1bf4ac336fec91defe77adb5e35abd074ee361fa02ee7d0722083e9fedbb4a0f WatchSource:0}: Error finding container 1bf4ac336fec91defe77adb5e35abd074ee361fa02ee7d0722083e9fedbb4a0f: Status 404 returned error can't find the container with id 1bf4ac336fec91defe77adb5e35abd074ee361fa02ee7d0722083e9fedbb4a0f
Nov 02 18:15:00.934134 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.934089    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"]
Nov 02 18:15:00.935100 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:15:00.935065    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef0830a_21d5_4002_9bfe_473d9bcc40c5.slice/crio-6bbbb4c977d947b9be0c96a1656102463602778f4be16e6012e07a004735c8c7 WatchSource:0}: Error finding container 6bbbb4c977d947b9be0c96a1656102463602778f4be16e6012e07a004735c8c7: Status 404 returned error can't find the container with id 6bbbb4c977d947b9be0c96a1656102463602778f4be16e6012e07a004735c8c7
Nov 02 18:15:00.977208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.977180    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"]
Nov 02 18:15:00.977664 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:15:00.977639    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430b80c8_354f_4332_b6c7_86c604facaaf.slice/crio-de85b4e54ccd1eb356f5d59ecd59e0c411f239c7d57ed91d29617ccd67728053 WatchSource:0}: Error finding container de85b4e54ccd1eb356f5d59ecd59e0c411f239c7d57ed91d29617ccd67728053: Status 404 returned error can't find the container with id de85b4e54ccd1eb356f5d59ecd59e0c411f239c7d57ed91d29617ccd67728053
Nov 02 18:15:00.980110 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:15:00.980084    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0058bed3_f1a1_4db6_8e8e_95a81f3d500b.slice/crio-b6356d3339278e40226f024fcbcd2e6c5fe871d967ce97aa7a38acfa8baa2f14 WatchSource:0}: Error finding container b6356d3339278e40226f024fcbcd2e6c5fe871d967ce97aa7a38acfa8baa2f14: Status 404 returned error can't find the container with id b6356d3339278e40226f024fcbcd2e6c5fe871d967ce97aa7a38acfa8baa2f14
Nov 02 18:15:00.980365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:00.980338    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"]
Nov 02 18:15:01.223339 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:01.223297    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" event={"ID":"fef0830a-21d5-4002-9bfe-473d9bcc40c5","Type":"ContainerStarted","Data":"6bbbb4c977d947b9be0c96a1656102463602778f4be16e6012e07a004735c8c7"}
Nov 02 18:15:01.224859 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:01.224827    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" event={"ID":"430b80c8-354f-4332-b6c7-86c604facaaf","Type":"ContainerStarted","Data":"de85b4e54ccd1eb356f5d59ecd59e0c411f239c7d57ed91d29617ccd67728053"}
Nov 02 18:15:01.229468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:01.227407    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" event={"ID":"0058bed3-f1a1-4db6-8e8e-95a81f3d500b","Type":"ContainerStarted","Data":"b6356d3339278e40226f024fcbcd2e6c5fe871d967ce97aa7a38acfa8baa2f14"}
Nov 02 18:15:01.229468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:01.228972    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" event={"ID":"d2c4b46d-ff39-476f-8582-06ddbf12afb5","Type":"ContainerStarted","Data":"1bf4ac336fec91defe77adb5e35abd074ee361fa02ee7d0722083e9fedbb4a0f"}
Nov 02 18:15:02.238057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.238024    4956 generic.go:356] "Generic (PLEG): container finished" podID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerID="8f930bfdea5cbbe19f3fc5f53c21f79c21e6c35dfe75219e771665e167d0ce26" exitCode=60
Nov 02 18:15:02.238566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.238108    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" event={"ID":"d2c4b46d-ff39-476f-8582-06ddbf12afb5","Type":"ContainerDied","Data":"8f930bfdea5cbbe19f3fc5f53c21f79c21e6c35dfe75219e771665e167d0ce26"}
Nov 02 18:15:02.238566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.238456    4956 scope.go:122] "RemoveContainer" containerID="8f930bfdea5cbbe19f3fc5f53c21f79c21e6c35dfe75219e771665e167d0ce26"
Nov 02 18:15:02.240583 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.240464    4956 generic.go:356] "Generic (PLEG): container finished" podID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerID="ef74f21cb476649e57a17570051c0894a4eff4f454571d174de4d778ebe533e2" exitCode=60
Nov 02 18:15:02.240583 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.240539    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" event={"ID":"fef0830a-21d5-4002-9bfe-473d9bcc40c5","Type":"ContainerDied","Data":"ef74f21cb476649e57a17570051c0894a4eff4f454571d174de4d778ebe533e2"}
Nov 02 18:15:02.240832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.240816    4956 scope.go:122] "RemoveContainer" containerID="ef74f21cb476649e57a17570051c0894a4eff4f454571d174de4d778ebe533e2"
Nov 02 18:15:02.242959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.242933    4956 generic.go:356] "Generic (PLEG): container finished" podID="430b80c8-354f-4332-b6c7-86c604facaaf" containerID="79915da8f6136d34068f85b8b1670852b338a3651322d08077ba5a7ca9d7ab8f" exitCode=60
Nov 02 18:15:02.243050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.243023    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" event={"ID":"430b80c8-354f-4332-b6c7-86c604facaaf","Type":"ContainerDied","Data":"79915da8f6136d34068f85b8b1670852b338a3651322d08077ba5a7ca9d7ab8f"}
Nov 02 18:15:02.243324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.243305    4956 scope.go:122] "RemoveContainer" containerID="79915da8f6136d34068f85b8b1670852b338a3651322d08077ba5a7ca9d7ab8f"
Nov 02 18:15:02.244848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.244828    4956 generic.go:356] "Generic (PLEG): container finished" podID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerID="15c0fd701101c56230eaa28807f6cb8621380a561a04b600424687258e5321c1" exitCode=60
Nov 02 18:15:02.244926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.244906    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" event={"ID":"0058bed3-f1a1-4db6-8e8e-95a81f3d500b","Type":"ContainerDied","Data":"15c0fd701101c56230eaa28807f6cb8621380a561a04b600424687258e5321c1"}
Nov 02 18:15:02.245097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:02.245080    4956 scope.go:122] "RemoveContainer" containerID="15c0fd701101c56230eaa28807f6cb8621380a561a04b600424687258e5321c1"
Nov 02 18:15:03.253466 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.253433    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" event={"ID":"fef0830a-21d5-4002-9bfe-473d9bcc40c5","Type":"ContainerStarted","Data":"d051d9140b526a3a2bf894449b2b114565f7e20dc59ac56eccadaa3481bc66b8"}
Nov 02 18:15:03.255416 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.255382    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" event={"ID":"430b80c8-354f-4332-b6c7-86c604facaaf","Type":"ContainerStarted","Data":"15c03168ad2d705def755f7f129087319e6418cbf501f0a77ba3db581a65c059"}
Nov 02 18:15:03.257463 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.257441    4956 generic.go:356] "Generic (PLEG): container finished" podID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerID="a2fc94a76e8e46df45d9bdd4eca2cd80149a664d27983044a0d6aabb3a74a58b" exitCode=60
Nov 02 18:15:03.257587 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.257524    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" event={"ID":"0058bed3-f1a1-4db6-8e8e-95a81f3d500b","Type":"ContainerDied","Data":"a2fc94a76e8e46df45d9bdd4eca2cd80149a664d27983044a0d6aabb3a74a58b"}
Nov 02 18:15:03.257587 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.257576    4956 scope.go:122] "RemoveContainer" containerID="15c0fd701101c56230eaa28807f6cb8621380a561a04b600424687258e5321c1"
Nov 02 18:15:03.257819 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.257804    4956 scope.go:122] "RemoveContainer" containerID="a2fc94a76e8e46df45d9bdd4eca2cd80149a664d27983044a0d6aabb3a74a58b"
Nov 02 18:15:03.258133 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:03.258087    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f_odh-ai-gateway-infra(0058bed3-f1a1-4db6-8e8e-95a81f3d500b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b"
Nov 02 18:15:03.259806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.259782    4956 generic.go:356] "Generic (PLEG): container finished" podID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerID="e38336ed93f2ed4455a5c75521d0bc4dc9f922b63046ed0b2b37b470f50e6664" exitCode=60
Nov 02 18:15:03.259900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.259832    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" event={"ID":"d2c4b46d-ff39-476f-8582-06ddbf12afb5","Type":"ContainerDied","Data":"e38336ed93f2ed4455a5c75521d0bc4dc9f922b63046ed0b2b37b470f50e6664"}
Nov 02 18:15:03.260057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.260045    4956 scope.go:122] "RemoveContainer" containerID="e38336ed93f2ed4455a5c75521d0bc4dc9f922b63046ed0b2b37b470f50e6664"
Nov 02 18:15:03.260320 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:03.260304    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl_odh-ai-gateway-infra(d2c4b46d-ff39-476f-8582-06ddbf12afb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5"
Nov 02 18:15:03.295255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.295230    4956 scope.go:122] "RemoveContainer" containerID="8f930bfdea5cbbe19f3fc5f53c21f79c21e6c35dfe75219e771665e167d0ce26"
Nov 02 18:15:03.366336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.366290    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" podStartSLOduration=3.36627471 podStartE2EDuration="3.36627471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.365332366 +0000 UTC m=+3133.622362985" watchObservedRunningTime="2026-11-02 18:15:03.36627471 +0000 UTC m=+3133.623305371"
Nov 02 18:15:03.704055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:03.704002    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" podStartSLOduration=3.70398738 podStartE2EDuration="3.70398738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.700633059 +0000 UTC m=+3133.957663678" watchObservedRunningTime="2026-11-02 18:15:03.70398738 +0000 UTC m=+3133.961017998"
Nov 02 18:15:04.268019 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.267984    4956 generic.go:356] "Generic (PLEG): container finished" podID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerID="d051d9140b526a3a2bf894449b2b114565f7e20dc59ac56eccadaa3481bc66b8" exitCode=60
Nov 02 18:15:04.268502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.268049    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" event={"ID":"fef0830a-21d5-4002-9bfe-473d9bcc40c5","Type":"ContainerDied","Data":"d051d9140b526a3a2bf894449b2b114565f7e20dc59ac56eccadaa3481bc66b8"}
Nov 02 18:15:04.268502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.268162    4956 scope.go:122] "RemoveContainer" containerID="ef74f21cb476649e57a17570051c0894a4eff4f454571d174de4d778ebe533e2"
Nov 02 18:15:04.268591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.268510    4956 scope.go:122] "RemoveContainer" containerID="d051d9140b526a3a2bf894449b2b114565f7e20dc59ac56eccadaa3481bc66b8"
Nov 02 18:15:04.268871 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:04.268840    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm_odh-ai-gateway-infra(fef0830a-21d5-4002-9bfe-473d9bcc40c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5"
Nov 02 18:15:04.270824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.270737    4956 generic.go:356] "Generic (PLEG): container finished" podID="430b80c8-354f-4332-b6c7-86c604facaaf" containerID="15c03168ad2d705def755f7f129087319e6418cbf501f0a77ba3db581a65c059" exitCode=60
Nov 02 18:15:04.270975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.270778    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" event={"ID":"430b80c8-354f-4332-b6c7-86c604facaaf","Type":"ContainerDied","Data":"15c03168ad2d705def755f7f129087319e6418cbf501f0a77ba3db581a65c059"}
Nov 02 18:15:04.271262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.271214    4956 scope.go:122] "RemoveContainer" containerID="15c03168ad2d705def755f7f129087319e6418cbf501f0a77ba3db581a65c059"
Nov 02 18:15:04.271551 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:04.271532    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68_odh-ai-gateway-infra(430b80c8-354f-4332-b6c7-86c604facaaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" podUID="430b80c8-354f-4332-b6c7-86c604facaaf"
Nov 02 18:15:04.273218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.273201    4956 scope.go:122] "RemoveContainer" containerID="a2fc94a76e8e46df45d9bdd4eca2cd80149a664d27983044a0d6aabb3a74a58b"
Nov 02 18:15:04.273514 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:04.273479    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f_odh-ai-gateway-infra(0058bed3-f1a1-4db6-8e8e-95a81f3d500b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b"
Nov 02 18:15:04.274925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.274909    4956 scope.go:122] "RemoveContainer" containerID="e38336ed93f2ed4455a5c75521d0bc4dc9f922b63046ed0b2b37b470f50e6664"
Nov 02 18:15:04.275174 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:04.275154    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl_odh-ai-gateway-infra(d2c4b46d-ff39-476f-8582-06ddbf12afb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5"
Nov 02 18:15:04.282721 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:04.282691    4956 scope.go:122] "RemoveContainer" containerID="79915da8f6136d34068f85b8b1670852b338a3651322d08077ba5a7ca9d7ab8f"
Nov 02 18:15:05.285270 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:05.285242    4956 scope.go:122] "RemoveContainer" containerID="d051d9140b526a3a2bf894449b2b114565f7e20dc59ac56eccadaa3481bc66b8"
Nov 02 18:15:05.285750 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:05.285601    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm_odh-ai-gateway-infra(fef0830a-21d5-4002-9bfe-473d9bcc40c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5"
Nov 02 18:15:05.286815 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:05.286792    4956 scope.go:122] "RemoveContainer" containerID="15c03168ad2d705def755f7f129087319e6418cbf501f0a77ba3db581a65c059"
Nov 02 18:15:05.287088 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:05.287067    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68_odh-ai-gateway-infra(430b80c8-354f-4332-b6c7-86c604facaaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" podUID="430b80c8-354f-4332-b6c7-86c604facaaf"
Nov 02 18:15:08.206484 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:08.206457    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:15:14.479724 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.479687    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"]
Nov 02 18:15:14.485356 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.485335    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.540869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.540819    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"]
Nov 02 18:15:14.585805 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.585770    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-3dc055-serving-cert\""
Nov 02 18:15:14.631613 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.631581    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-openshift-service-ca\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.631789 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.631764    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.631910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.631892    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvd9b\" (UniqueName: \"kubernetes.io/projected/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-kube-api-access-dvd9b\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.632026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.632007    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-metrics-certs\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.733819 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.733713    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-openshift-service-ca\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.733982 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.733841    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.733982 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.733920    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvd9b\" (UniqueName: \"kubernetes.io/projected/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-kube-api-access-dvd9b\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.734158 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.734006    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-metrics-certs\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.734158 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:14.734094    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-serving-cert: secret "maas-api-e2e-explicit-3dc055-serving-cert" not found
Nov 02 18:15:14.734268 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:14.734188    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls podName:5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7 nodeName:}" failed. No retries permitted until 2026-11-02 18:15:15.234165954 +0000 UTC m=+3145.491196551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls") pod "maas-api-e2e-explicit-3dc055-f4d54488d-jq42c" (UID: "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7") : secret "maas-api-e2e-explicit-3dc055-serving-cert" not found
Nov 02 18:15:14.734694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.734670    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-openshift-service-ca\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.737109 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.737080    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-metrics-certs\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:14.791712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:14.791682    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvd9b\" (UniqueName: \"kubernetes.io/projected/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-kube-api-access-dvd9b\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:15.239222 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.239185    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:15.241799 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.241778    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls\") pod \"maas-api-e2e-explicit-3dc055-f4d54488d-jq42c\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:15.398621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.398581    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:15.776993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.776963    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"]
Nov 02 18:15:15.777875 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:15:15.777850    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0ab2c5_9bef_47e6_bff5_d645f81bd8f7.slice/crio-9c5d005e4711f974ca6e49f3fcd493f99e7f1b3f9e7fbd8bceb5a39903590ec1 WatchSource:0}: Error finding container 9c5d005e4711f974ca6e49f3fcd493f99e7f1b3f9e7fbd8bceb5a39903590ec1: Status 404 returned error can't find the container with id 9c5d005e4711f974ca6e49f3fcd493f99e7f1b3f9e7fbd8bceb5a39903590ec1
Nov 02 18:15:15.779595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.779573    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:15:15.858278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.858249    4956 scope.go:122] "RemoveContainer" containerID="a2fc94a76e8e46df45d9bdd4eca2cd80149a664d27983044a0d6aabb3a74a58b"
Nov 02 18:15:15.858475 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.858363    4956 scope.go:122] "RemoveContainer" containerID="15c03168ad2d705def755f7f129087319e6418cbf501f0a77ba3db581a65c059"
Nov 02 18:15:15.858525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:15.858497    4956 scope.go:122] "RemoveContainer" containerID="d051d9140b526a3a2bf894449b2b114565f7e20dc59ac56eccadaa3481bc66b8"
Nov 02 18:15:16.356251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:16.356209    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c" event={"ID":"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7","Type":"ContainerStarted","Data":"9c5d005e4711f974ca6e49f3fcd493f99e7f1b3f9e7fbd8bceb5a39903590ec1"}
Nov 02 18:15:16.358261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:16.358230    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" event={"ID":"fef0830a-21d5-4002-9bfe-473d9bcc40c5","Type":"ContainerStarted","Data":"cd58ffcdfd10a05de44d9047e45dd390643d6b38af553b4aed1b30396a1cb8a0"}
Nov 02 18:15:16.360434 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:16.360409    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" event={"ID":"430b80c8-354f-4332-b6c7-86c604facaaf","Type":"ContainerStarted","Data":"a746cdc0a639b0991f371768ca6fec9b228a2de7b42908e977bab66f8f9c9f0b"}
Nov 02 18:15:16.857792 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:16.857567    4956 scope.go:122] "RemoveContainer" containerID="e38336ed93f2ed4455a5c75521d0bc4dc9f922b63046ed0b2b37b470f50e6664"
Nov 02 18:15:17.369932 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.369901    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" event={"ID":"d2c4b46d-ff39-476f-8582-06ddbf12afb5","Type":"ContainerStarted","Data":"eddd7af67c57f976eae6976bd63d4148bd69783fe93db985a3a4535408594839"}
Nov 02 18:15:17.371691 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.371660    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c" event={"ID":"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7","Type":"ContainerStarted","Data":"b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92"}
Nov 02 18:15:17.371842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.371719    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:17.373427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.373398    4956 generic.go:356] "Generic (PLEG): container finished" podID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerID="cd58ffcdfd10a05de44d9047e45dd390643d6b38af553b4aed1b30396a1cb8a0" exitCode=60
Nov 02 18:15:17.373528 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.373440    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" event={"ID":"fef0830a-21d5-4002-9bfe-473d9bcc40c5","Type":"ContainerDied","Data":"cd58ffcdfd10a05de44d9047e45dd390643d6b38af553b4aed1b30396a1cb8a0"}
Nov 02 18:15:17.373528 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.373498    4956 scope.go:122] "RemoveContainer" containerID="d051d9140b526a3a2bf894449b2b114565f7e20dc59ac56eccadaa3481bc66b8"
Nov 02 18:15:17.373754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.373729    4956 scope.go:122] "RemoveContainer" containerID="cd58ffcdfd10a05de44d9047e45dd390643d6b38af553b4aed1b30396a1cb8a0"
Nov 02 18:15:17.373990 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:17.373965    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm_odh-ai-gateway-infra(fef0830a-21d5-4002-9bfe-473d9bcc40c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5"
Nov 02 18:15:17.375412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.375393    4956 generic.go:356] "Generic (PLEG): container finished" podID="430b80c8-354f-4332-b6c7-86c604facaaf" containerID="a746cdc0a639b0991f371768ca6fec9b228a2de7b42908e977bab66f8f9c9f0b" exitCode=60
Nov 02 18:15:17.375499 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.375456    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" event={"ID":"430b80c8-354f-4332-b6c7-86c604facaaf","Type":"ContainerDied","Data":"a746cdc0a639b0991f371768ca6fec9b228a2de7b42908e977bab66f8f9c9f0b"}
Nov 02 18:15:17.375750 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.375732    4956 scope.go:122] "RemoveContainer" containerID="a746cdc0a639b0991f371768ca6fec9b228a2de7b42908e977bab66f8f9c9f0b"
Nov 02 18:15:17.375962 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:17.375945    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68_odh-ai-gateway-infra(430b80c8-354f-4332-b6c7-86c604facaaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" podUID="430b80c8-354f-4332-b6c7-86c604facaaf"
Nov 02 18:15:17.377550 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.377528    4956 generic.go:356] "Generic (PLEG): container finished" podID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerID="34c5d4ec9d54102472e850f867cc73c834f4a28d7c7d3859ccfe3ecfaff6d915" exitCode=60
Nov 02 18:15:17.377629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.377571    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" event={"ID":"0058bed3-f1a1-4db6-8e8e-95a81f3d500b","Type":"ContainerDied","Data":"34c5d4ec9d54102472e850f867cc73c834f4a28d7c7d3859ccfe3ecfaff6d915"}
Nov 02 18:15:17.377832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.377819    4956 scope.go:122] "RemoveContainer" containerID="34c5d4ec9d54102472e850f867cc73c834f4a28d7c7d3859ccfe3ecfaff6d915"
Nov 02 18:15:17.378008 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:17.377992    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f_odh-ai-gateway-infra(0058bed3-f1a1-4db6-8e8e-95a81f3d500b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b"
Nov 02 18:15:17.386389 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.386371    4956 scope.go:122] "RemoveContainer" containerID="15c03168ad2d705def755f7f129087319e6418cbf501f0a77ba3db581a65c059"
Nov 02 18:15:17.399640 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.399620    4956 scope.go:122] "RemoveContainer" containerID="a2fc94a76e8e46df45d9bdd4eca2cd80149a664d27983044a0d6aabb3a74a58b"
Nov 02 18:15:17.482132 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.482087    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"]
Nov 02 18:15:17.489192 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.489107    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" podStartSLOduration=17.489089033 podStartE2EDuration="17.489089033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:17.488777545 +0000 UTC m=+3147.745808165" watchObservedRunningTime="2026-11-02 18:15:17.489089033 +0000 UTC m=+3147.746119655"
Nov 02 18:15:17.595280 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.595183    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"]
Nov 02 18:15:17.828574 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:17.828517    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c" podStartSLOduration=2.850351684 podStartE2EDuration="3.828500663s" totalImagesPullingTime="978.148979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:17.827189568 +0000 UTC m=+3148.084220190" watchObservedRunningTime="2026-11-02 18:15:17.828500663 +0000 UTC m=+3148.085531281"
Nov 02 18:15:18.385263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.385231    4956 generic.go:356] "Generic (PLEG): container finished" podID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerID="eddd7af67c57f976eae6976bd63d4148bd69783fe93db985a3a4535408594839" exitCode=60
Nov 02 18:15:18.385737 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.385276    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" event={"ID":"d2c4b46d-ff39-476f-8582-06ddbf12afb5","Type":"ContainerDied","Data":"eddd7af67c57f976eae6976bd63d4148bd69783fe93db985a3a4535408594839"}
Nov 02 18:15:18.385737 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.385337    4956 scope.go:122] "RemoveContainer" containerID="e38336ed93f2ed4455a5c75521d0bc4dc9f922b63046ed0b2b37b470f50e6664"
Nov 02 18:15:18.385856 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.385765    4956 scope.go:122] "RemoveContainer" containerID="eddd7af67c57f976eae6976bd63d4148bd69783fe93db985a3a4535408594839"
Nov 02 18:15:18.386077 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:18.386054    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl_odh-ai-gateway-infra(d2c4b46d-ff39-476f-8582-06ddbf12afb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5"
Nov 02 18:15:18.507047 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.507011    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"]
Nov 02 18:15:18.586004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.585980    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:18.588509 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.588487    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:18.686461 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.686349    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430b80c8-354f-4332-b6c7-86c604facaaf-kube-api-access-7gwzx\" (UniqueName: \"kubernetes.io/projected/430b80c8-354f-4332-b6c7-86c604facaaf-kube-api-access-7gwzx\") pod \"430b80c8-354f-4332-b6c7-86c604facaaf\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") "
Nov 02 18:15:18.686626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.686463    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef0830a-21d5-4002-9bfe-473d9bcc40c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef0830a-21d5-4002-9bfe-473d9bcc40c5-maas-api-service-ca\") pod \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") "
Nov 02 18:15:18.686626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.686485    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430b80c8-354f-4332-b6c7-86c604facaaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430b80c8-354f-4332-b6c7-86c604facaaf-maas-api-service-ca\") pod \"430b80c8-354f-4332-b6c7-86c604facaaf\" (UID: \"430b80c8-354f-4332-b6c7-86c604facaaf\") "
Nov 02 18:15:18.686626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.686523    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef0830a-21d5-4002-9bfe-473d9bcc40c5-kube-api-access-xj2lw\" (UniqueName: \"kubernetes.io/projected/fef0830a-21d5-4002-9bfe-473d9bcc40c5-kube-api-access-xj2lw\") pod \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\" (UID: \"fef0830a-21d5-4002-9bfe-473d9bcc40c5\") "
Nov 02 18:15:18.686856 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.686831    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef0830a-21d5-4002-9bfe-473d9bcc40c5-maas-api-service-ca" pod "fef0830a-21d5-4002-9bfe-473d9bcc40c5" (UID: "fef0830a-21d5-4002-9bfe-473d9bcc40c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:18.686908 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.686867    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430b80c8-354f-4332-b6c7-86c604facaaf-maas-api-service-ca" pod "430b80c8-354f-4332-b6c7-86c604facaaf" (UID: "430b80c8-354f-4332-b6c7-86c604facaaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:18.687174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.687156    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef0830a-21d5-4002-9bfe-473d9bcc40c5-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:18.687216 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.687177    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/430b80c8-354f-4332-b6c7-86c604facaaf-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:18.688831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.688815    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef0830a-21d5-4002-9bfe-473d9bcc40c5-kube-api-access-xj2lw" pod "fef0830a-21d5-4002-9bfe-473d9bcc40c5" (UID: "fef0830a-21d5-4002-9bfe-473d9bcc40c5"). InnerVolumeSpecName "kube-api-access-xj2lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:18.688888 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.688841    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430b80c8-354f-4332-b6c7-86c604facaaf-kube-api-access-7gwzx" pod "430b80c8-354f-4332-b6c7-86c604facaaf" (UID: "430b80c8-354f-4332-b6c7-86c604facaaf"). InnerVolumeSpecName "kube-api-access-7gwzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:18.788236 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.788207    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gwzx\" (UniqueName: \"kubernetes.io/projected/430b80c8-354f-4332-b6c7-86c604facaaf-kube-api-access-7gwzx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:18.788236 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.788233    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2lw\" (UniqueName: \"kubernetes.io/projected/fef0830a-21d5-4002-9bfe-473d9bcc40c5-kube-api-access-xj2lw\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:18.953532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:18.953499    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"]
Nov 02 18:15:19.131171 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.131143    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:19.193394 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.193366    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-kube-api-access-p2sgw\" (UniqueName: \"kubernetes.io/projected/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-kube-api-access-p2sgw\") pod \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") "
Nov 02 18:15:19.193569 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.193492    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-maas-api-service-ca\") pod \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\" (UID: \"0058bed3-f1a1-4db6-8e8e-95a81f3d500b\") "
Nov 02 18:15:19.193824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.193802    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-maas-api-service-ca" pod "0058bed3-f1a1-4db6-8e8e-95a81f3d500b" (UID: "0058bed3-f1a1-4db6-8e8e-95a81f3d500b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:19.193995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.193983    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:19.195861 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.195838    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-kube-api-access-p2sgw" pod "0058bed3-f1a1-4db6-8e8e-95a81f3d500b" (UID: "0058bed3-f1a1-4db6-8e8e-95a81f3d500b"). InnerVolumeSpecName "kube-api-access-p2sgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:19.295330 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.295300    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2sgw\" (UniqueName: \"kubernetes.io/projected/0058bed3-f1a1-4db6-8e8e-95a81f3d500b-kube-api-access-p2sgw\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:19.399042 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.399008    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"
Nov 02 18:15:19.399566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.399016    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm" event={"ID":"fef0830a-21d5-4002-9bfe-473d9bcc40c5","Type":"ContainerDied","Data":"6bbbb4c977d947b9be0c96a1656102463602778f4be16e6012e07a004735c8c7"}
Nov 02 18:15:19.399566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.399205    4956 scope.go:122] "RemoveContainer" containerID="cd58ffcdfd10a05de44d9047e45dd390643d6b38af553b4aed1b30396a1cb8a0"
Nov 02 18:15:19.400506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.400486    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"
Nov 02 18:15:19.400582 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.400502    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68" event={"ID":"430b80c8-354f-4332-b6c7-86c604facaaf","Type":"ContainerDied","Data":"de85b4e54ccd1eb356f5d59ecd59e0c411f239c7d57ed91d29617ccd67728053"}
Nov 02 18:15:19.402160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.402108    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"
Nov 02 18:15:19.402302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.402155    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f" event={"ID":"0058bed3-f1a1-4db6-8e8e-95a81f3d500b","Type":"ContainerDied","Data":"b6356d3339278e40226f024fcbcd2e6c5fe871d967ce97aa7a38acfa8baa2f14"}
Nov 02 18:15:19.427314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.427286    4956 scope.go:122] "RemoveContainer" containerID="a746cdc0a639b0991f371768ca6fec9b228a2de7b42908e977bab66f8f9c9f0b"
Nov 02 18:15:19.437315 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.437285    4956 scope.go:122] "RemoveContainer" containerID="34c5d4ec9d54102472e850f867cc73c834f4a28d7c7d3859ccfe3ecfaff6d915"
Nov 02 18:15:19.564608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.564582    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:19.566896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.566874    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"]
Nov 02 18:15:19.571677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.571654    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894055-zvs8f"]
Nov 02 18:15:19.598966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.598941    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c4b46d-ff39-476f-8582-06ddbf12afb5-kube-api-access-dmwhz\" (UniqueName: \"kubernetes.io/projected/d2c4b46d-ff39-476f-8582-06ddbf12afb5-kube-api-access-dmwhz\") pod \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") "
Nov 02 18:15:19.599104 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.599067    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c4b46d-ff39-476f-8582-06ddbf12afb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c4b46d-ff39-476f-8582-06ddbf12afb5-maas-api-service-ca\") pod \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\" (UID: \"d2c4b46d-ff39-476f-8582-06ddbf12afb5\") "
Nov 02 18:15:19.599417 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.599395    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c4b46d-ff39-476f-8582-06ddbf12afb5-maas-api-service-ca" pod "d2c4b46d-ff39-476f-8582-06ddbf12afb5" (UID: "d2c4b46d-ff39-476f-8582-06ddbf12afb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:19.601161 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.601138    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c4b46d-ff39-476f-8582-06ddbf12afb5-kube-api-access-dmwhz" pod "d2c4b46d-ff39-476f-8582-06ddbf12afb5" (UID: "d2c4b46d-ff39-476f-8582-06ddbf12afb5"). InnerVolumeSpecName "kube-api-access-dmwhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:19.700972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.700940    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmwhz\" (UniqueName: \"kubernetes.io/projected/d2c4b46d-ff39-476f-8582-06ddbf12afb5-kube-api-access-dmwhz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:19.700972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.700974    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c4b46d-ff39-476f-8582-06ddbf12afb5-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:19.788635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.788547    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"]
Nov 02 18:15:19.791544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.791521    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894055-bg9xm"]
Nov 02 18:15:19.865499 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.865472    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" path="/var/lib/kubelet/pods/0058bed3-f1a1-4db6-8e8e-95a81f3d500b/volumes"
Nov 02 18:15:19.865869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.865856    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" path="/var/lib/kubelet/pods/fef0830a-21d5-4002-9bfe-473d9bcc40c5/volumes"
Nov 02 18:15:19.900350 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.900321    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"]
Nov 02 18:15:19.904496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:19.904473    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894055-xjc68"]
Nov 02 18:15:20.415289 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:20.415256    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"
Nov 02 18:15:20.415767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:20.415298    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl" event={"ID":"d2c4b46d-ff39-476f-8582-06ddbf12afb5","Type":"ContainerDied","Data":"1bf4ac336fec91defe77adb5e35abd074ee361fa02ee7d0722083e9fedbb4a0f"}
Nov 02 18:15:20.415767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:20.415361    4956 scope.go:122] "RemoveContainer" containerID="eddd7af67c57f976eae6976bd63d4148bd69783fe93db985a3a4535408594839"
Nov 02 18:15:20.581947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:20.581915    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"]
Nov 02 18:15:20.587160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:20.587106    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894055-k4vfl"]
Nov 02 18:15:21.865477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:21.865444    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" path="/var/lib/kubelet/pods/430b80c8-354f-4332-b6c7-86c604facaaf/volumes"
Nov 02 18:15:21.866002 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:21.865986    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" path="/var/lib/kubelet/pods/d2c4b46d-ff39-476f-8582-06ddbf12afb5/volumes"
Nov 02 18:15:23.397600 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:23.397570    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:15:29.052535 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:29.052496    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:15:57.848693 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:57.848652    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"]
Nov 02 18:15:57.849331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:15:57.848918    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" containerID="cri-o://b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4" gracePeriod=30
Nov 02 18:15:59.053724 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:15:59.053687    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:16:01.859261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:01.859159    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:16:01.859261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:01.859244    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:16:03.126639 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.126577    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:16:03.136822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.136793    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-token\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-token\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.136994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.136835    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-socket\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.136994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.136906    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-envoy\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.136994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.136949    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-podinfo\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.136994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.136971    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-certs\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.137252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137012    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-kube-api-access-mftvj\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-kube-api-access-mftvj\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.137252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137063    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5094140-37a0-4338-8033-be2df9a9a3c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5094140-37a0-4338-8033-be2df9a9a3c8-istiod-ca-cert\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.137252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137166    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-socket" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:03.137252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137201    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-credential-socket\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.137468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137268    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-data\") pod \"b5094140-37a0-4338-8033-be2df9a9a3c8\" (UID: \"b5094140-37a0-4338-8033-be2df9a9a3c8\") "
Nov 02 18:16:03.137468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137274    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-certs" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:03.137665 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137472    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-credential-socket" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:03.137665 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137487    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-data" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:03.137665 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137534    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5094140-37a0-4338-8033-be2df9a9a3c8-istiod-ca-cert" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:03.138009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.137985    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.138090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.138014    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5094140-37a0-4338-8033-be2df9a9a3c8-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.138090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.138032    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.138090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.138046    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.138090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.138061    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.139266 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.139242    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-token" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:03.139347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.139272    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-envoy" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:03.139347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.139326    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-kube-api-access-mftvj" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "kube-api-access-mftvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:03.139735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.139714    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-podinfo" pod "b5094140-37a0-4338-8033-be2df9a9a3c8" (UID: "b5094140-37a0-4338-8033-be2df9a9a3c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:16:03.239172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.239107    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.239172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.239167    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.239172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.239178    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5094140-37a0-4338-8033-be2df9a9a3c8-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.239417 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.239189    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftvj\" (UniqueName: \"kubernetes.io/projected/b5094140-37a0-4338-8033-be2df9a9a3c8-kube-api-access-mftvj\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:03.703238 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.703198    4956 generic.go:356] "Generic (PLEG): container finished" podID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerID="b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4" exitCode=0
Nov 02 18:16:03.703562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.703258    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"
Nov 02 18:16:03.703562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.703286    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" event={"ID":"b5094140-37a0-4338-8033-be2df9a9a3c8","Type":"ContainerDied","Data":"b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4"}
Nov 02 18:16:03.703562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.703360    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk" event={"ID":"b5094140-37a0-4338-8033-be2df9a9a3c8","Type":"ContainerDied","Data":"20c39d4ca5a46e85178381d9dfa5a3f467e3ab9525d6ee0be2a5d9edf864a3a5"}
Nov 02 18:16:03.703562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.703393    4956 scope.go:122] "RemoveContainer" containerID="b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4"
Nov 02 18:16:03.715057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.715036    4956 scope.go:122] "RemoveContainer" containerID="b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4"
Nov 02 18:16:03.715388 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:16:03.715368    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4\": container with ID starting with b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4 not found: ID does not exist" containerID="b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4"
Nov 02 18:16:03.715448 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.715398    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4"} err="failed to get container status \"b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4\": rpc error: code = NotFound desc = could not find container \"b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4\": container with ID starting with b34727e411ae1be3bedfccf4b6026e5008de780c9509bb597ba346a463ec24f4 not found: ID does not exist"
Nov 02 18:16:03.870787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.870754    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"]
Nov 02 18:16:03.873623 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:03.873602    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-6bpwk"]
Nov 02 18:16:05.866608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:16:05.866562    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" path="/var/lib/kubelet/pods/b5094140-37a0-4338-8033-be2df9a9a3c8/volumes"
Nov 02 18:16:29.053251 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:16:29.053206    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:16:59.052453 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:16:59.052419    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:17:11.427690 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:11.427651    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"]
Nov 02 18:17:11.428320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:11.427856    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" containerID="cri-o://8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc" gracePeriod=30
Nov 02 18:17:15.001146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:15.001095    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:17:15.001542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:15.001178    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:17:16.711818 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.711786    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:17:16.738936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.738902    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-podinfo\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739109 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.738967    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-certs\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739109 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739060    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-kube-api-access-llb6c\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-kube-api-access-llb6c\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739177    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-credential-socket\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739210    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istiod-ca-cert\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739230    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-socket\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739265    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-certs" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:16.739505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739282    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-token\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739301    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-envoy\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739323    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-data\") pod \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\" (UID: \"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01\") "
Nov 02 18:17:16.739505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739439    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-credential-socket" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:16.739505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739487    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-socket" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:16.739931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739836    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.739931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739853    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.739931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739862    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.739931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739865    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istiod-ca-cert" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:17:16.739931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.739886    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-data" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:16.741411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.741385    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-podinfo" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:17:16.741515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.741448    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-kube-api-access-llb6c" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "kube-api-access-llb6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:16.741554 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.741529    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-token" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:16.741889 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.741866    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-envoy" pod "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" (UID: "b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:16.840590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.840552    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.840590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.840584    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llb6c\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-kube-api-access-llb6c\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.840590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.840595    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.840828 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.840604    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.840828 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.840613    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:16.840828 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:16.840621    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:17.166807 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.166719    4956 generic.go:356] "Generic (PLEG): container finished" podID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerID="8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc" exitCode=0
Nov 02 18:17:17.166807 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.166775    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"
Nov 02 18:17:17.167016 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.166801    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" event={"ID":"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01","Type":"ContainerDied","Data":"8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc"}
Nov 02 18:17:17.167016 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.166863    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl" event={"ID":"b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01","Type":"ContainerDied","Data":"280e025dc20568864933c958c579f3afe2c8436411827fe51ded53d3905ecbcc"}
Nov 02 18:17:17.167016 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.166895    4956 scope.go:122] "RemoveContainer" containerID="8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc"
Nov 02 18:17:17.177711 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.177686    4956 scope.go:122] "RemoveContainer" containerID="8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc"
Nov 02 18:17:17.178040 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:17.178019    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc\": container with ID starting with 8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc not found: ID does not exist" containerID="8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc"
Nov 02 18:17:17.178101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.178049    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc"} err="failed to get container status \"8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc\": rpc error: code = NotFound desc = could not find container \"8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc\": container with ID starting with 8be2139ebdc39c4fe4ddf53fc398d6ff7762d403a2930d67bca5600ed29808fc not found: ID does not exist"
Nov 02 18:17:17.331319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.331285    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"]
Nov 02 18:17:17.336304 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.336271    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-f45rl"]
Nov 02 18:17:17.865261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:17.865228    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" path="/var/lib/kubelet/pods/b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01/volumes"
Nov 02 18:17:29.054167 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:29.054101    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:17:38.805652 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.805610    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7526-0c7f3c"]
Nov 02 18:17:38.806236 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806222    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:38.806236 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806237    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806249    4956 cpu_manager.go:414] "containerMap: removing container" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806254    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806268    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806273    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806294    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806298    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806305    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806310    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806316    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806320    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806327    4956 cpu_manager.go:414] "containerMap: removing container" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:38.806337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806331    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806344    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806350    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806360    4956 cpu_manager.go:414] "containerMap: removing container" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806364    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806372    4956 cpu_manager.go:414] "containerMap: removing container" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806376    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806482    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806490    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806495    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806500    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c4aa2e-ef7c-4dd0-b39f-5ac417993b01" containerName="istio-proxy"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806505    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806512    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5094140-37a0-4338-8033-be2df9a9a3c8" containerName="istio-proxy"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806517    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806523    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806528    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806535    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806541    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:38.806694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.806547    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:38.810011 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.809995    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:38.863183 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.863145    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7526-0c7f3c"]
Nov 02 18:17:38.899454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.899417    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcczm\" (UniqueName: \"kubernetes.io/projected/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252-kube-api-access-hcczm\") pod \"test-curl-7526-0c7f3c\" (UID: \"22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252\") " pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:38.907730 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:38.907703    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4gwc\""
Nov 02 18:17:39.001083 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:39.001031    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcczm\" (UniqueName: \"kubernetes.io/projected/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252-kube-api-access-hcczm\") pod \"test-curl-7526-0c7f3c\" (UID: \"22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252\") " pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:39.058556 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:39.058480    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcczm\" (UniqueName: \"kubernetes.io/projected/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252-kube-api-access-hcczm\") pod \"test-curl-7526-0c7f3c\" (UID: \"22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252\") " pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:39.123040 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:39.123005    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:39.496222 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:39.496192    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7526-0c7f3c"]
Nov 02 18:17:39.497843 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:17:39.497816    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22e7ae33_0c1e_4d63_8b8d_fb73e0a5d252.slice/crio-7cb4417a0b34f7cf9d3f66fb3e085d5000d50265076ee63f8b3c2fcfeb8fb734 WatchSource:0}: Error finding container 7cb4417a0b34f7cf9d3f66fb3e085d5000d50265076ee63f8b3c2fcfeb8fb734: Status 404 returned error can't find the container with id 7cb4417a0b34f7cf9d3f66fb3e085d5000d50265076ee63f8b3c2fcfeb8fb734
Nov 02 18:17:40.111180 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111145    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-c141b3"]
Nov 02 18:17:40.111744 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111729    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:40.111792 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111745    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:40.111792 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111769    4956 cpu_manager.go:414] "containerMap: removing container" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:40.111792 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111776    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0058bed3-f1a1-4db6-8e8e-95a81f3d500b" containerName="cleanup"
Nov 02 18:17:40.111891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111792    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:40.111891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111798    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:40.111891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111805    4956 cpu_manager.go:414] "containerMap: removing container" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:40.111891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111810    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430b80c8-354f-4332-b6c7-86c604facaaf" containerName="cleanup"
Nov 02 18:17:40.112011 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111899    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c4b46d-ff39-476f-8582-06ddbf12afb5" containerName="cleanup"
Nov 02 18:17:40.112011 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.111908    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef0830a-21d5-4002-9bfe-473d9bcc40c5" containerName="cleanup"
Nov 02 18:17:40.116403 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.116386    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:40.169526 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.169494    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-c141b3"]
Nov 02 18:17:40.226234 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.226194    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjws\" (UniqueName: \"kubernetes.io/projected/52d380c9-c307-4241-9da0-65f611d10027-kube-api-access-pvjws\") pod \"test-curl-7520-c141b3\" (UID: \"52d380c9-c307-4241-9da0-65f611d10027\") " pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:40.307165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.307130    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7526-0c7f3c" event={"ID":"22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252","Type":"ContainerStarted","Data":"7cb4417a0b34f7cf9d3f66fb3e085d5000d50265076ee63f8b3c2fcfeb8fb734"}
Nov 02 18:17:40.327999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.327974    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjws\" (UniqueName: \"kubernetes.io/projected/52d380c9-c307-4241-9da0-65f611d10027-kube-api-access-pvjws\") pod \"test-curl-7520-c141b3\" (UID: \"52d380c9-c307-4241-9da0-65f611d10027\") " pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:40.384870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.384798    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjws\" (UniqueName: \"kubernetes.io/projected/52d380c9-c307-4241-9da0-65f611d10027-kube-api-access-pvjws\") pod \"test-curl-7520-c141b3\" (UID: \"52d380c9-c307-4241-9da0-65f611d10027\") " pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:40.429376 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.429342    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:40.806891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:40.806859    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-c141b3"]
Nov 02 18:17:40.808414 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:17:40.808384    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52d380c9_c307_4241_9da0_65f611d10027.slice/crio-6fe6d75f0370278e6ff94e8bafc304359558cbc5902a6e44ea340dc288f6aff8 WatchSource:0}: Error finding container 6fe6d75f0370278e6ff94e8bafc304359558cbc5902a6e44ea340dc288f6aff8: Status 404 returned error can't find the container with id 6fe6d75f0370278e6ff94e8bafc304359558cbc5902a6e44ea340dc288f6aff8
Nov 02 18:17:41.316105 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:41.316068    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-c141b3" event={"ID":"52d380c9-c307-4241-9da0-65f611d10027","Type":"ContainerStarted","Data":"6fe6d75f0370278e6ff94e8bafc304359558cbc5902a6e44ea340dc288f6aff8"}
Nov 02 18:17:42.325219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:42.325172    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7526-0c7f3c" event={"ID":"22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252","Type":"ContainerStarted","Data":"3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918"}
Nov 02 18:17:42.436815 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:42.436712    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7526-0c7f3c" podStartSLOduration=1.838940589 podStartE2EDuration="4.436692874s" totalImagesPullingTime="2.597752285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:42.435147565 +0000 UTC m=+3292.692178185" watchObservedRunningTime="2026-11-02 18:17:42.436692874 +0000 UTC m=+3292.693723497"
Nov 02 18:17:42.878752 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:42.878722    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"]
Nov 02 18:17:43.333451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:43.333414    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-c141b3" event={"ID":"52d380c9-c307-4241-9da0-65f611d10027","Type":"ContainerStarted","Data":"4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9"}
Nov 02 18:17:43.445757 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:43.445701    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7520-c141b3" podStartSLOduration=1.776617934 podStartE2EDuration="3.445684238s" totalImagesPullingTime="1.669066304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:43.444765684 +0000 UTC m=+3293.701796307" watchObservedRunningTime="2026-11-02 18:17:43.445684238 +0000 UTC m=+3293.702714858"
Nov 02 18:17:43.879893 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:43.879860    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7526-0c7f3c"]
Nov 02 18:17:43.882871 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:43.882840    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7526-0c7f3c"]
Nov 02 18:17:44.340365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:44.340318    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7526-0c7f3c" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" containerName="test-curl-7526-0c7f3c" containerID="cri-o://3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918" gracePeriod=2
Nov 02 18:17:44.390922 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:44.390877    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7526-0c7f3c\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:44.914820 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:44.914782    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-c141b3"]
Nov 02 18:17:44.919749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:44.919717    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-c141b3"]
Nov 02 18:17:45.311018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.310983    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-1218ea"]
Nov 02 18:17:45.311585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.311571    4956 cpu_manager.go:414] "containerMap: removing container" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" containerName="test-curl-7526-0c7f3c"
Nov 02 18:17:45.311633 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.311587    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" containerName="test-curl-7526-0c7f3c"
Nov 02 18:17:45.311633 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.311612    4956 cpu_manager.go:414] "containerMap: removing container" podUID="52d380c9-c307-4241-9da0-65f611d10027" containerName="test-curl-7520-c141b3"
Nov 02 18:17:45.311633 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.311617    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d380c9-c307-4241-9da0-65f611d10027" containerName="test-curl-7520-c141b3"
Nov 02 18:17:45.311731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.311720    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" containerName="test-curl-7526-0c7f3c"
Nov 02 18:17:45.311764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.311732    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d380c9-c307-4241-9da0-65f611d10027" containerName="test-curl-7520-c141b3"
Nov 02 18:17:45.315110 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.315091    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:45.346305 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.346245    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7520-c141b3" podUID="52d380c9-c307-4241-9da0-65f611d10027" containerName="test-curl-7520-c141b3" containerID="cri-o://4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9" gracePeriod=2
Nov 02 18:17:45.365886 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:45.365844    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7526-0c7f3c\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:45.368611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.368578    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-1218ea"]
Nov 02 18:17:45.393778 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.393742    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t722q\" (UniqueName: \"kubernetes.io/projected/6389315a-4f5a-4a1a-a947-91e3ba24d9c3-kube-api-access-t722q\") pod \"test-curl-7520-1218ea\" (UID: \"6389315a-4f5a-4a1a-a947-91e3ba24d9c3\") " pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:45.494556 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.494521    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t722q\" (UniqueName: \"kubernetes.io/projected/6389315a-4f5a-4a1a-a947-91e3ba24d9c3-kube-api-access-t722q\") pod \"test-curl-7520-1218ea\" (UID: \"6389315a-4f5a-4a1a-a947-91e3ba24d9c3\") " pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:45.525344 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:45.525305    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-c141b3\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="52d380c9-c307-4241-9da0-65f611d10027" pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:45.550737 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.550698    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t722q\" (UniqueName: \"kubernetes.io/projected/6389315a-4f5a-4a1a-a947-91e3ba24d9c3-kube-api-access-t722q\") pod \"test-curl-7520-1218ea\" (UID: \"6389315a-4f5a-4a1a-a947-91e3ba24d9c3\") " pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:45.575936 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:45.575843    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7526-0c7f3c\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:45.627309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:45.627277    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:46.004340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.004310    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-1218ea"]
Nov 02 18:17:46.005214 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:17:46.005182    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6389315a_4f5a_4a1a_a947_91e3ba24d9c3.slice/crio-e4bb6a0fee31ed8bb86438f62bd18e8b0e6ec9439fc63b1c820567e2294e0401 WatchSource:0}: Error finding container e4bb6a0fee31ed8bb86438f62bd18e8b0e6ec9439fc63b1c820567e2294e0401: Status 404 returned error can't find the container with id e4bb6a0fee31ed8bb86438f62bd18e8b0e6ec9439fc63b1c820567e2294e0401
Nov 02 18:17:46.354377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.354341    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-1218ea" event={"ID":"6389315a-4f5a-4a1a-a947-91e3ba24d9c3","Type":"ContainerStarted","Data":"fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb"}
Nov 02 18:17:46.354708 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.354398    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-1218ea" event={"ID":"6389315a-4f5a-4a1a-a947-91e3ba24d9c3","Type":"ContainerStarted","Data":"e4bb6a0fee31ed8bb86438f62bd18e8b0e6ec9439fc63b1c820567e2294e0401"}
Nov 02 18:17:46.466102 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.466049    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7520-1218ea" podStartSLOduration=1.466033172 podStartE2EDuration="1.466033172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:46.463904049 +0000 UTC m=+3296.720934671" watchObservedRunningTime="2026-11-02 18:17:46.466033172 +0000 UTC m=+3296.723063791"
Nov 02 18:17:46.582861 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.582839    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:46.609302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.609236    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252-kube-api-access-hcczm\" (UniqueName: \"kubernetes.io/projected/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252-kube-api-access-hcczm\") pod \"22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252\" (UID: \"22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252\") "
Nov 02 18:17:46.611551 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.611527    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252-kube-api-access-hcczm" pod "22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" (UID: "22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252"). InnerVolumeSpecName "kube-api-access-hcczm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:46.711377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:46.711347    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcczm\" (UniqueName: \"kubernetes.io/projected/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252-kube-api-access-hcczm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:47.361994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.361965    4956 generic.go:356] "Generic (PLEG): container finished" podID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" containerID="3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918" exitCode=137
Nov 02 18:17:47.362483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.362013    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:47.362483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.362062    4956 scope.go:122] "RemoveContainer" containerID="3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918"
Nov 02 18:17:47.413870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.413844    4956 scope.go:122] "RemoveContainer" containerID="3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918"
Nov 02 18:17:47.414322 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:47.414291    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918\": container with ID starting with 3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918 not found: ID does not exist" containerID="3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918"
Nov 02 18:17:47.414450 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.414342    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918"} err="failed to get container status \"3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918\": rpc error: code = NotFound desc = could not find container \"3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918\": container with ID starting with 3b7ed6cd4b6b67de24dcce3baf6c53a95b5f0dcd29eaf4acba3a0cd15f692918 not found: ID does not exist"
Nov 02 18:17:47.459702 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:47.459665    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7526-0c7f3c\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:47.603554 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.603482    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:47.654169 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:47.654114    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7526-0c7f3c\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" pod="opendatahub/test-curl-7526-0c7f3c"
Nov 02 18:17:47.729300 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.729267    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d380c9-c307-4241-9da0-65f611d10027-kube-api-access-pvjws\" (UniqueName: \"kubernetes.io/projected/52d380c9-c307-4241-9da0-65f611d10027-kube-api-access-pvjws\") pod \"52d380c9-c307-4241-9da0-65f611d10027\" (UID: \"52d380c9-c307-4241-9da0-65f611d10027\") "
Nov 02 18:17:47.731693 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.731662    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d380c9-c307-4241-9da0-65f611d10027-kube-api-access-pvjws" pod "52d380c9-c307-4241-9da0-65f611d10027" (UID: "52d380c9-c307-4241-9da0-65f611d10027"). InnerVolumeSpecName "kube-api-access-pvjws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:47.830969 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.830936    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjws\" (UniqueName: \"kubernetes.io/projected/52d380c9-c307-4241-9da0-65f611d10027-kube-api-access-pvjws\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:47.845683 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.845651    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-1218ea"]
Nov 02 18:17:47.850875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.850845    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-1218ea"]
Nov 02 18:17:47.866564 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.866483    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252" path="/var/lib/kubelet/pods/22e7ae33-0c1e-4d63-8b8d-fb73e0a5d252/volumes"
Nov 02 18:17:47.866867 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.866852    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d380c9-c307-4241-9da0-65f611d10027" path="/var/lib/kubelet/pods/52d380c9-c307-4241-9da0-65f611d10027/volumes"
Nov 02 18:17:47.995438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.995402    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-8fabfe"]
Nov 02 18:17:47.996027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.996012    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6389315a-4f5a-4a1a-a947-91e3ba24d9c3" containerName="test-curl-7520-1218ea"
Nov 02 18:17:47.996075 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.996030    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6389315a-4f5a-4a1a-a947-91e3ba24d9c3" containerName="test-curl-7520-1218ea"
Nov 02 18:17:47.996169 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.996159    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6389315a-4f5a-4a1a-a947-91e3ba24d9c3" containerName="test-curl-7520-1218ea"
Nov 02 18:17:47.999775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:47.999760    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:48.032830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.032797    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7w6l\" (UniqueName: \"kubernetes.io/projected/6b1a4a8e-23a4-4ce1-bba0-a26da836b237-kube-api-access-t7w6l\") pod \"test-curl-7520-8fabfe\" (UID: \"6b1a4a8e-23a4-4ce1-bba0-a26da836b237\") " pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:48.053405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.053371    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-8fabfe"]
Nov 02 18:17:48.133859 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.133775    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7w6l\" (UniqueName: \"kubernetes.io/projected/6b1a4a8e-23a4-4ce1-bba0-a26da836b237-kube-api-access-t7w6l\") pod \"test-curl-7520-8fabfe\" (UID: \"6b1a4a8e-23a4-4ce1-bba0-a26da836b237\") " pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:48.191522 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.191490    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7w6l\" (UniqueName: \"kubernetes.io/projected/6b1a4a8e-23a4-4ce1-bba0-a26da836b237-kube-api-access-t7w6l\") pod \"test-curl-7520-8fabfe\" (UID: \"6b1a4a8e-23a4-4ce1-bba0-a26da836b237\") " pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:48.312445 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.312405    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:48.373313 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.373276    4956 generic.go:356] "Generic (PLEG): container finished" podID="52d380c9-c307-4241-9da0-65f611d10027" containerID="4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9" exitCode=137
Nov 02 18:17:48.373742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.373330    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-c141b3"
Nov 02 18:17:48.373742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.373367    4956 scope.go:122] "RemoveContainer" containerID="4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9"
Nov 02 18:17:48.373742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.373544    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7520-1218ea" podUID="6389315a-4f5a-4a1a-a947-91e3ba24d9c3" containerName="test-curl-7520-1218ea" containerID="cri-o://fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb" gracePeriod=2
Nov 02 18:17:48.392396 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.392357    4956 scope.go:122] "RemoveContainer" containerID="4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9"
Nov 02 18:17:48.393341 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:48.393315    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9\": container with ID starting with 4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9 not found: ID does not exist" containerID="4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9"
Nov 02 18:17:48.393505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.393348    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9"} err="failed to get container status \"4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9\": rpc error: code = NotFound desc = could not find container \"4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9\": container with ID starting with 4aa7a10557edf4678df449917c3750f233e6a344612c0e70ce16e150530c29e9 not found: ID does not exist"
Nov 02 18:17:48.687857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:48.687825    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-8fabfe"]
Nov 02 18:17:48.689462 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:17:48.689429    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b1a4a8e_23a4_4ce1_bba0_a26da836b237.slice/crio-b66e6fe840c84b283b260a4cdf919106074a88544bc4a3ba1d9abf3815210a85 WatchSource:0}: Error finding container b66e6fe840c84b283b260a4cdf919106074a88544bc4a3ba1d9abf3815210a85: Status 404 returned error can't find the container with id b66e6fe840c84b283b260a4cdf919106074a88544bc4a3ba1d9abf3815210a85
Nov 02 18:17:49.381902 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:49.381860    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-8fabfe" event={"ID":"6b1a4a8e-23a4-4ce1-bba0-a26da836b237","Type":"ContainerStarted","Data":"34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7"}
Nov 02 18:17:49.382364 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:49.381912    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-8fabfe" event={"ID":"6b1a4a8e-23a4-4ce1-bba0-a26da836b237","Type":"ContainerStarted","Data":"b66e6fe840c84b283b260a4cdf919106074a88544bc4a3ba1d9abf3815210a85"}
Nov 02 18:17:49.496221 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:49.496169    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7520-8fabfe" podStartSLOduration=2.496155129 podStartE2EDuration="2.496155129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:49.492791142 +0000 UTC m=+3299.749821776" watchObservedRunningTime="2026-11-02 18:17:49.496155129 +0000 UTC m=+3299.753185748"
Nov 02 18:17:50.310072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.310034    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:17:50.310646 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.310625    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:17:50.648787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.648761    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:50.766668 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.766633    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6389315a-4f5a-4a1a-a947-91e3ba24d9c3-kube-api-access-t722q\" (UniqueName: \"kubernetes.io/projected/6389315a-4f5a-4a1a-a947-91e3ba24d9c3-kube-api-access-t722q\") pod \"6389315a-4f5a-4a1a-a947-91e3ba24d9c3\" (UID: \"6389315a-4f5a-4a1a-a947-91e3ba24d9c3\") "
Nov 02 18:17:50.768842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.768817    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6389315a-4f5a-4a1a-a947-91e3ba24d9c3-kube-api-access-t722q" pod "6389315a-4f5a-4a1a-a947-91e3ba24d9c3" (UID: "6389315a-4f5a-4a1a-a947-91e3ba24d9c3"). InnerVolumeSpecName "kube-api-access-t722q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:50.868411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.868328    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t722q\" (UniqueName: \"kubernetes.io/projected/6389315a-4f5a-4a1a-a947-91e3ba24d9c3-kube-api-access-t722q\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:50.945872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.945830    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-8fabfe"]
Nov 02 18:17:50.950360 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.950330    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-8fabfe"]
Nov 02 18:17:50.998997 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.998960    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"]
Nov 02 18:17:50.999634 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.999618    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" containerName="test-curl-7520-8fabfe"
Nov 02 18:17:50.999693 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.999636    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" containerName="test-curl-7520-8fabfe"
Nov 02 18:17:50.999780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:50.999771    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" containerName="test-curl-7520-8fabfe"
Nov 02 18:17:51.004555 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.004534    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.059309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.059275    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"]
Nov 02 18:17:51.071157 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.071102    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1cecd5-11db-4379-b6c4-289351da9614-tls-certs\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.071371 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.071274    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-tmp-dir\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.071371 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.071310    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmdn\" (UniqueName: \"kubernetes.io/projected/dd1cecd5-11db-4379-b6c4-289351da9614-kube-api-access-jfmdn\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.071502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.071427    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-model-cache\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.071502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.071482    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-home\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.071609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.071585    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-dshm\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.104352 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.104314    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-eb709f\"/\"openshift-service-ca.crt\""
Nov 02 18:17:51.104352 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.104325    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-eb709f\"/\"kube-root-ca.crt\""
Nov 02 18:17:51.104352 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.104363    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-eb709f\"/\"autores-model-eb709f-kserve-self-signed-certs\""
Nov 02 18:17:51.105659 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.105641    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-eb709f\"/\"default-dockercfg-rnbv8\""
Nov 02 18:17:51.173418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173328    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-tmp-dir\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.173418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173374    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmdn\" (UniqueName: \"kubernetes.io/projected/dd1cecd5-11db-4379-b6c4-289351da9614-kube-api-access-jfmdn\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.173660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173440    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-model-cache\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.173660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173478    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-home\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.173660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173515    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-dshm\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.173660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173632    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1cecd5-11db-4379-b6c4-289351da9614-tls-certs\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.173866 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173769    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-tmp-dir\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.173866 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173780    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-model-cache\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.174002 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.173967    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-home\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.176018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.175996    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-dshm\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.176467 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.176444    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1cecd5-11db-4379-b6c4-289351da9614-tls-certs\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.230278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.230247    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmdn\" (UniqueName: \"kubernetes.io/projected/dd1cecd5-11db-4379-b6c4-289351da9614-kube-api-access-jfmdn\") pod \"autores-model-eb709f-kserve-69bf85555c-drwzs\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") " pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.319598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.319558    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:51.401085 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.401046    4956 generic.go:356] "Generic (PLEG): container finished" podID="6389315a-4f5a-4a1a-a947-91e3ba24d9c3" containerID="fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb" exitCode=137
Nov 02 18:17:51.401293 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.401264    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7520-8fabfe" podUID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" containerName="test-curl-7520-8fabfe" containerID="cri-o://34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7" gracePeriod=2
Nov 02 18:17:51.401681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.401661    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:51.402373 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.401991    4956 scope.go:122] "RemoveContainer" containerID="fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb"
Nov 02 18:17:51.425771 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.423198    4956 scope.go:122] "RemoveContainer" containerID="fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb"
Nov 02 18:17:51.425771 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:51.423585    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb\": container with ID starting with fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb not found: ID does not exist" containerID="fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb"
Nov 02 18:17:51.425771 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.423616    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb"} err="failed to get container status \"fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb\": rpc error: code = NotFound desc = could not find container \"fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb\": container with ID starting with fc32bc525d5764f8e22ffe38ab7dae327c2ee0b255b2dc19e79dbd930719dfdb not found: ID does not exist"
Nov 02 18:17:51.452803 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:51.452752    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-8fabfe\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:51.503660 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:51.503619    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-1218ea\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="6389315a-4f5a-4a1a-a947-91e3ba24d9c3" pod="opendatahub/test-curl-7520-1218ea"
Nov 02 18:17:51.553965 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:51.553926    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-8fabfe\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:51.697209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.697182    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"]
Nov 02 18:17:51.698461 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:17:51.698432    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd1cecd5_11db_4379_b6c4_289351da9614.slice/crio-e2abd3b63906c8f9655fb2ef6d1c743845f9627262a47e70da3386d41aff6bfd WatchSource:0}: Error finding container e2abd3b63906c8f9655fb2ef6d1c743845f9627262a47e70da3386d41aff6bfd: Status 404 returned error can't find the container with id e2abd3b63906c8f9655fb2ef6d1c743845f9627262a47e70da3386d41aff6bfd
Nov 02 18:17:51.866059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:51.866018    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6389315a-4f5a-4a1a-a947-91e3ba24d9c3" path="/var/lib/kubelet/pods/6389315a-4f5a-4a1a-a947-91e3ba24d9c3/volumes"
Nov 02 18:17:52.408947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:52.408909    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs" event={"ID":"dd1cecd5-11db-4379-b6c4-289351da9614","Type":"ContainerStarted","Data":"78083e0cd82c773b14f4198c467c4f354a938567c32a5c8183a446c1b2865b42"}
Nov 02 18:17:52.408947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:52.408953    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs" event={"ID":"dd1cecd5-11db-4379-b6c4-289351da9614","Type":"ContainerStarted","Data":"e2abd3b63906c8f9655fb2ef6d1c743845f9627262a47e70da3386d41aff6bfd"}
Nov 02 18:17:52.522983 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:52.522899    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs" podStartSLOduration=2.522878171 podStartE2EDuration="2.522878171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:52.521641836 +0000 UTC m=+3302.778672463" watchObservedRunningTime="2026-11-02 18:17:52.522878171 +0000 UTC m=+3302.779908794"
Nov 02 18:17:53.320214 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:53.320179    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:17:53.652743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:53.652714    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:53.711211 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:53.711175    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1a4a8e-23a4-4ce1-bba0-a26da836b237-kube-api-access-t7w6l\" (UniqueName: \"kubernetes.io/projected/6b1a4a8e-23a4-4ce1-bba0-a26da836b237-kube-api-access-t7w6l\") pod \"6b1a4a8e-23a4-4ce1-bba0-a26da836b237\" (UID: \"6b1a4a8e-23a4-4ce1-bba0-a26da836b237\") "
Nov 02 18:17:53.713457 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:53.713433    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1a4a8e-23a4-4ce1-bba0-a26da836b237-kube-api-access-t7w6l" pod "6b1a4a8e-23a4-4ce1-bba0-a26da836b237" (UID: "6b1a4a8e-23a4-4ce1-bba0-a26da836b237"). InnerVolumeSpecName "kube-api-access-t7w6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:53.812569 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:53.812537    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7w6l\" (UniqueName: \"kubernetes.io/projected/6b1a4a8e-23a4-4ce1-bba0-a26da836b237-kube-api-access-t7w6l\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:53.865813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:53.865766    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" path="/var/lib/kubelet/pods/6b1a4a8e-23a4-4ce1-bba0-a26da836b237/volumes"
Nov 02 18:17:54.436903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:54.436867    4956 generic.go:356] "Generic (PLEG): container finished" podID="6b1a4a8e-23a4-4ce1-bba0-a26da836b237" containerID="34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7" exitCode=137
Nov 02 18:17:54.437411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:54.436937    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-8fabfe"
Nov 02 18:17:54.437411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:54.436957    4956 scope.go:122] "RemoveContainer" containerID="34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7"
Nov 02 18:17:54.446782 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:54.446766    4956 scope.go:122] "RemoveContainer" containerID="34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7"
Nov 02 18:17:54.447179 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:54.447152    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7\": container with ID starting with 34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7 not found: ID does not exist" containerID="34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7"
Nov 02 18:17:54.447248 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:54.447201    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7"} err="failed to get container status \"34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7\": rpc error: code = NotFound desc = could not find container \"34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7\": container with ID starting with 34378e138777836f8f56f83313a5140bd5ce53a59474ebcbf29c5910e5b76ca7 not found: ID does not exist"
Nov 02 18:17:57.900408 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:57.900370    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b" podUID="f6332051-1697-415c-92dd-abfdcd40f827" containerName="main" containerID="cri-o://0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678" gracePeriod=45
Nov 02 18:17:58.181479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.181452    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:17:58.270493 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270461    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-dshm\") pod \"f6332051-1697-415c-92dd-abfdcd40f827\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") "
Nov 02 18:17:58.270493 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270497    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-tmp-dir\") pod \"f6332051-1697-415c-92dd-abfdcd40f827\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") "
Nov 02 18:17:58.270728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270523    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6332051-1697-415c-92dd-abfdcd40f827-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6332051-1697-415c-92dd-abfdcd40f827-tls-certs\") pod \"f6332051-1697-415c-92dd-abfdcd40f827\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") "
Nov 02 18:17:58.270728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270567    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-model-cache\") pod \"f6332051-1697-415c-92dd-abfdcd40f827\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") "
Nov 02 18:17:58.270728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270641    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-home\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-home\") pod \"f6332051-1697-415c-92dd-abfdcd40f827\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") "
Nov 02 18:17:58.270728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270689    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6332051-1697-415c-92dd-abfdcd40f827-kube-api-access-zmp26\" (UniqueName: \"kubernetes.io/projected/f6332051-1697-415c-92dd-abfdcd40f827-kube-api-access-zmp26\") pod \"f6332051-1697-415c-92dd-abfdcd40f827\" (UID: \"f6332051-1697-415c-92dd-abfdcd40f827\") "
Nov 02 18:17:58.270933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270813    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-tmp-dir" pod "f6332051-1697-415c-92dd-abfdcd40f827" (UID: "f6332051-1697-415c-92dd-abfdcd40f827"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:58.270933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.270894    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-home" pod "f6332051-1697-415c-92dd-abfdcd40f827" (UID: "f6332051-1697-415c-92dd-abfdcd40f827"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:58.271202 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.271173    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-model-cache" pod "f6332051-1697-415c-92dd-abfdcd40f827" (UID: "f6332051-1697-415c-92dd-abfdcd40f827"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:58.271555 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.271518    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.271662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.271574    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.271662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.271591    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.273155 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.273104    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6332051-1697-415c-92dd-abfdcd40f827-tls-certs" pod "f6332051-1697-415c-92dd-abfdcd40f827" (UID: "f6332051-1697-415c-92dd-abfdcd40f827"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:58.273274 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.273254    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-dshm" pod "f6332051-1697-415c-92dd-abfdcd40f827" (UID: "f6332051-1697-415c-92dd-abfdcd40f827"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:58.273339 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.273297    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6332051-1697-415c-92dd-abfdcd40f827-kube-api-access-zmp26" pod "f6332051-1697-415c-92dd-abfdcd40f827" (UID: "f6332051-1697-415c-92dd-abfdcd40f827"). InnerVolumeSpecName "kube-api-access-zmp26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:58.372870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.372834    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6332051-1697-415c-92dd-abfdcd40f827-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.372870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.372861    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6332051-1697-415c-92dd-abfdcd40f827-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.372870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.372877    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmp26\" (UniqueName: \"kubernetes.io/projected/f6332051-1697-415c-92dd-abfdcd40f827-kube-api-access-zmp26\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.466093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.466064    4956 generic.go:356] "Generic (PLEG): container finished" podID="f6332051-1697-415c-92dd-abfdcd40f827" containerID="0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678" exitCode=0
Nov 02 18:17:58.466259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.466128    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"
Nov 02 18:17:58.466259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.466161    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b" event={"ID":"f6332051-1697-415c-92dd-abfdcd40f827","Type":"ContainerDied","Data":"0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678"}
Nov 02 18:17:58.466259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.466206    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b" event={"ID":"f6332051-1697-415c-92dd-abfdcd40f827","Type":"ContainerDied","Data":"5b1c70a72a53728996f5b8bca3b650fe74fc53850de22ff3b98983aad50fcb1b"}
Nov 02 18:17:58.466259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.466229    4956 scope.go:122] "RemoveContainer" containerID="0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678"
Nov 02 18:17:58.477294 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.477274    4956 scope.go:122] "RemoveContainer" containerID="0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678"
Nov 02 18:17:58.477543 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:58.477525    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678\": container with ID starting with 0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678 not found: ID does not exist" containerID="0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678"
Nov 02 18:17:58.477591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.477551    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678"} err="failed to get container status \"0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678\": rpc error: code = NotFound desc = could not find container \"0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678\": container with ID starting with 0ce2c8d4dd8635d42fb0ef6a0f46483eb9b656725162ff20ca31b34b2a635678 not found: ID does not exist"
Nov 02 18:17:58.630201 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.630167    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"]
Nov 02 18:17:58.633592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:58.633567    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4ecad8/auth-test-model-4ecad8-kserve-5b74c6745-q8x4b"]
Nov 02 18:17:59.052475 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:17:59.052429    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:17:59.866266 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:17:59.866229    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6332051-1697-415c-92dd-abfdcd40f827" path="/var/lib/kubelet/pods/f6332051-1697-415c-92dd-abfdcd40f827/volumes"
Nov 02 18:18:01.319826 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:01.319788    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:18:01.333025 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:01.332995    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:18:01.498647 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:01.498613    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:18:09.301068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.301038    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"]
Nov 02 18:18:09.301735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.301718    4956 cpu_manager.go:414] "containerMap: removing container" podUID="f6332051-1697-415c-92dd-abfdcd40f827" containerName="main"
Nov 02 18:18:09.301780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.301737    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6332051-1697-415c-92dd-abfdcd40f827" containerName="main"
Nov 02 18:18:09.301880 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.301870    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6332051-1697-415c-92dd-abfdcd40f827" containerName="main"
Nov 02 18:18:09.306576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.306560    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.361997 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.361961    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"]
Nov 02 18:18:09.408478 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.408440    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a8da6e\""
Nov 02 18:18:09.408678 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.408500    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a8da6e-dockercfg-mqpkt\""
Nov 02 18:18:09.409157 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.409101    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa577e5-a13b-4d2b-b610-582bf0ed01d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.409483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.409459    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d997\" (UniqueName: \"kubernetes.io/projected/faa577e5-a13b-4d2b-b610-582bf0ed01d5-kube-api-access-6d997\") pod \"payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.467665 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.467618    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"]
Nov 02 18:18:09.471683 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.471663    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:09.510773 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.510736    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcccd037-a971-4c7f-974f-9f2b95da6fe2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:09.510970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.510795    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d997\" (UniqueName: \"kubernetes.io/projected/faa577e5-a13b-4d2b-b610-582bf0ed01d5-kube-api-access-6d997\") pod \"payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.510970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.510925    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trzs\" (UniqueName: \"kubernetes.io/projected/dcccd037-a971-4c7f-974f-9f2b95da6fe2-kube-api-access-8trzs\") pod \"payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:09.511062 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.510987    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa577e5-a13b-4d2b-b610-582bf0ed01d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.511695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.511677    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa577e5-a13b-4d2b-b610-582bf0ed01d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.528558 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.528529    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"]
Nov 02 18:18:09.568747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.568673    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d997\" (UniqueName: \"kubernetes.io/projected/faa577e5-a13b-4d2b-b610-582bf0ed01d5-kube-api-access-6d997\") pod \"payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.612706 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.612671    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trzs\" (UniqueName: \"kubernetes.io/projected/dcccd037-a971-4c7f-974f-9f2b95da6fe2-kube-api-access-8trzs\") pod \"payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:09.612910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.612816    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcccd037-a971-4c7f-974f-9f2b95da6fe2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:09.613491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.613472    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcccd037-a971-4c7f-974f-9f2b95da6fe2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:09.620371 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.620352    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:09.673808 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.673528    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trzs\" (UniqueName: \"kubernetes.io/projected/dcccd037-a971-4c7f-974f-9f2b95da6fe2-kube-api-access-8trzs\") pod \"payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:09.785945 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:09.785915    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:10.010249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.010224    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"]
Nov 02 18:18:10.010851 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:18:10.010823    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa577e5_a13b_4d2b_b610_582bf0ed01d5.slice/crio-53e0743da30301af74b61301f45ca0ed54b98f7e78453a6e3b355fa928645b68 WatchSource:0}: Error finding container 53e0743da30301af74b61301f45ca0ed54b98f7e78453a6e3b355fa928645b68: Status 404 returned error can't find the container with id 53e0743da30301af74b61301f45ca0ed54b98f7e78453a6e3b355fa928645b68
Nov 02 18:18:10.162056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.162010    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"]
Nov 02 18:18:10.546321 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.546290    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2" event={"ID":"faa577e5-a13b-4d2b-b610-582bf0ed01d5","Type":"ContainerStarted","Data":"23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365"}
Nov 02 18:18:10.546836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.546342    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2" event={"ID":"faa577e5-a13b-4d2b-b610-582bf0ed01d5","Type":"ContainerStarted","Data":"53e0743da30301af74b61301f45ca0ed54b98f7e78453a6e3b355fa928645b68"}
Nov 02 18:18:10.546836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.546510    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:10.548129 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.548097    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg" event={"ID":"dcccd037-a971-4c7f-974f-9f2b95da6fe2","Type":"ContainerStarted","Data":"dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663"}
Nov 02 18:18:10.548190 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.548164    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg" event={"ID":"dcccd037-a971-4c7f-974f-9f2b95da6fe2","Type":"ContainerStarted","Data":"25bfe28738d6be2f68452a96082e66fdb224c6a3378177e791a0fda7489434fa"}
Nov 02 18:18:10.548234 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.548193    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:10.662071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.662021    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2" podStartSLOduration=1.6620055200000001 podStartE2EDuration="1.66200552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:10.659363994 +0000 UTC m=+3320.916394617" watchObservedRunningTime="2026-11-02 18:18:10.66200552 +0000 UTC m=+3320.919036137"
Nov 02 18:18:10.773777 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:10.773705    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg" podStartSLOduration=1.773686247 podStartE2EDuration="1.773686247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:10.771881772 +0000 UTC m=+3321.028912392" watchObservedRunningTime="2026-11-02 18:18:10.773686247 +0000 UTC m=+3321.030716868"
Nov 02 18:18:21.560410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:21.560378    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:18:21.561468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:21.561448    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:18:29.053177 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:18:29.053139    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:18:31.043542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:31.043505    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"]
Nov 02 18:18:46.065053 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.064984    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr" podUID="8b83246f-76b0-477c-9eff-2f7800508bbd" containerName="main" containerID="cri-o://f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735" gracePeriod=45
Nov 02 18:18:46.326900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.326833    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:18:46.481717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.481682    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-model-cache\") pod \"8b83246f-76b0-477c-9eff-2f7800508bbd\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") "
Nov 02 18:18:46.481910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.481737    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b83246f-76b0-477c-9eff-2f7800508bbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b83246f-76b0-477c-9eff-2f7800508bbd-tls-certs\") pod \"8b83246f-76b0-477c-9eff-2f7800508bbd\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") "
Nov 02 18:18:46.481910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.481774    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b83246f-76b0-477c-9eff-2f7800508bbd-kube-api-access-7vflb\" (UniqueName: \"kubernetes.io/projected/8b83246f-76b0-477c-9eff-2f7800508bbd-kube-api-access-7vflb\") pod \"8b83246f-76b0-477c-9eff-2f7800508bbd\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") "
Nov 02 18:18:46.481910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.481795    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-home\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-home\") pod \"8b83246f-76b0-477c-9eff-2f7800508bbd\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") "
Nov 02 18:18:46.481910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.481843    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-tmp-dir\") pod \"8b83246f-76b0-477c-9eff-2f7800508bbd\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") "
Nov 02 18:18:46.482094 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.481937    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-dshm\") pod \"8b83246f-76b0-477c-9eff-2f7800508bbd\" (UID: \"8b83246f-76b0-477c-9eff-2f7800508bbd\") "
Nov 02 18:18:46.482094 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.482021    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-model-cache" pod "8b83246f-76b0-477c-9eff-2f7800508bbd" (UID: "8b83246f-76b0-477c-9eff-2f7800508bbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:46.482213 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.482191    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-home" pod "8b83246f-76b0-477c-9eff-2f7800508bbd" (UID: "8b83246f-76b0-477c-9eff-2f7800508bbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:46.482512 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.482478    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-tmp-dir" pod "8b83246f-76b0-477c-9eff-2f7800508bbd" (UID: "8b83246f-76b0-477c-9eff-2f7800508bbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:46.482609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.482565    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:46.482609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.482598    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:46.484614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.484587    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b83246f-76b0-477c-9eff-2f7800508bbd-kube-api-access-7vflb" pod "8b83246f-76b0-477c-9eff-2f7800508bbd" (UID: "8b83246f-76b0-477c-9eff-2f7800508bbd"). InnerVolumeSpecName "kube-api-access-7vflb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:46.484712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.484686    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-dshm" pod "8b83246f-76b0-477c-9eff-2f7800508bbd" (UID: "8b83246f-76b0-477c-9eff-2f7800508bbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:46.484849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.484829    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b83246f-76b0-477c-9eff-2f7800508bbd-tls-certs" pod "8b83246f-76b0-477c-9eff-2f7800508bbd" (UID: "8b83246f-76b0-477c-9eff-2f7800508bbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:18:46.584361 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.584271    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b83246f-76b0-477c-9eff-2f7800508bbd-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:46.584361 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.584299    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vflb\" (UniqueName: \"kubernetes.io/projected/8b83246f-76b0-477c-9eff-2f7800508bbd-kube-api-access-7vflb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:46.584361 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.584310    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:46.584361 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.584319    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b83246f-76b0-477c-9eff-2f7800508bbd-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:46.788108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.788076    4956 generic.go:356] "Generic (PLEG): container finished" podID="8b83246f-76b0-477c-9eff-2f7800508bbd" containerID="f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735" exitCode=0
Nov 02 18:18:46.788341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.788139    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"
Nov 02 18:18:46.788341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.788163    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr" event={"ID":"8b83246f-76b0-477c-9eff-2f7800508bbd","Type":"ContainerDied","Data":"f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735"}
Nov 02 18:18:46.788341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.788208    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr" event={"ID":"8b83246f-76b0-477c-9eff-2f7800508bbd","Type":"ContainerDied","Data":"ff0a22764b10cd42ba57a47cd0c6ac36e78417a27e4bdc72389cf17943058796"}
Nov 02 18:18:46.788341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.788231    4956 scope.go:122] "RemoveContainer" containerID="f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735"
Nov 02 18:18:46.802049 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.802018    4956 scope.go:122] "RemoveContainer" containerID="f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735"
Nov 02 18:18:46.802397 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:18:46.802377    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735\": container with ID starting with f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735 not found: ID does not exist" containerID="f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735"
Nov 02 18:18:46.802468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.802415    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735"} err="failed to get container status \"f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735\": rpc error: code = NotFound desc = could not find container \"f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735\": container with ID starting with f798d877d2a2971ea26be3a179e3527e779896ea5227bc2e3999865ffc2f7735 not found: ID does not exist"
Nov 02 18:18:46.952865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.952834    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"]
Nov 02 18:18:46.956576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:46.956548    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/auth-test-model-2a1bcc-kserve-b9b4b48df-vsjpr"]
Nov 02 18:18:47.873983 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:18:47.873947    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b83246f-76b0-477c-9eff-2f7800508bbd" path="/var/lib/kubelet/pods/8b83246f-76b0-477c-9eff-2f7800508bbd/volumes"
Nov 02 18:18:59.053826 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:18:59.053783    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:19:04.897924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:04.897841    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"]
Nov 02 18:19:04.898457 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:04.898088    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c" podUID="5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" containerName="maas-api" containerID="cri-o://b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92" gracePeriod=30
Nov 02 18:19:05.162166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.162080    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:19:05.302977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.302923    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-kube-api-access-dvd9b\" (UniqueName: \"kubernetes.io/projected/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-kube-api-access-dvd9b\") pod \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") "
Nov 02 18:19:05.303215 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.303062    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-metrics-certs\") pod \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") "
Nov 02 18:19:05.303306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.303230    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls\") pod \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") "
Nov 02 18:19:05.303306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.303293    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-openshift-service-ca\") pod \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\" (UID: \"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7\") "
Nov 02 18:19:05.304333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.304299    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-openshift-service-ca" pod "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" (UID: "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:19:05.305685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.305656    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-metrics-certs" pod "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" (UID: "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:05.306091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.306060    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-kube-api-access-dvd9b" pod "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" (UID: "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7"). InnerVolumeSpecName "kube-api-access-dvd9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:19:05.306563 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.306537    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls" pod "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" (UID: "5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:05.405084 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.405046    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:05.405084 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.405083    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:05.405330 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.405094    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:05.405330 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.405106    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvd9b\" (UniqueName: \"kubernetes.io/projected/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7-kube-api-access-dvd9b\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:05.907576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.907536    4956 generic.go:356] "Generic (PLEG): container finished" podID="5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" containerID="b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92" exitCode=0
Nov 02 18:19:05.908026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.907611    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"
Nov 02 18:19:05.908026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.907618    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c" event={"ID":"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7","Type":"ContainerDied","Data":"b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92"}
Nov 02 18:19:05.908026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.907686    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c" event={"ID":"5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7","Type":"ContainerDied","Data":"9c5d005e4711f974ca6e49f3fcd493f99e7f1b3f9e7fbd8bceb5a39903590ec1"}
Nov 02 18:19:05.908026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.907716    4956 scope.go:122] "RemoveContainer" containerID="b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92"
Nov 02 18:19:05.919444 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.919422    4956 scope.go:122] "RemoveContainer" containerID="b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92"
Nov 02 18:19:05.919737 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:19:05.919715    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92\": container with ID starting with b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92 not found: ID does not exist" containerID="b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92"
Nov 02 18:19:05.919791 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:05.919748    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92"} err="failed to get container status \"b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92\": rpc error: code = NotFound desc = could not find container \"b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92\": container with ID starting with b1607b0f4e892ab6d18b613ee0731c8b9775dd429ac9a3f7d098e39e88f02f92 not found: ID does not exist"
Nov 02 18:19:06.071203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:06.071170    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"]
Nov 02 18:19:06.120730 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:06.120693    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3dc055-f4d54488d-jq42c"]
Nov 02 18:19:07.865988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:07.865950    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" path="/var/lib/kubelet/pods/5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7/volumes"
Nov 02 18:19:08.863038 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.863009    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"]
Nov 02 18:19:08.863603 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.863590    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8b83246f-76b0-477c-9eff-2f7800508bbd" containerName="main"
Nov 02 18:19:08.863656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.863605    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b83246f-76b0-477c-9eff-2f7800508bbd" containerName="main"
Nov 02 18:19:08.863656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.863649    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" containerName="maas-api"
Nov 02 18:19:08.863656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.863654    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" containerName="maas-api"
Nov 02 18:19:08.863756 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.863748    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0ab2c5-9bef-47e6-bff5-d645f81bd8f7" containerName="maas-api"
Nov 02 18:19:08.863788 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.863759    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b83246f-76b0-477c-9eff-2f7800508bbd" containerName="main"
Nov 02 18:19:08.869114 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.869094    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.921442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.921410    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-bcda4b-openshift-default-dockercfg-l2t7g\""
Nov 02 18:19:08.949728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.949689    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-socket\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.949919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.949737    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-envoy\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.949919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.949810    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b065d348-957d-4e03-b951-eccd1952ec83-istiod-ca-cert\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.950027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.949945    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-credential-socket\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.950027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.949982    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-istio-token\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.950027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.950018    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-certs\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.950343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.950321    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4mb\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-kube-api-access-mk4mb\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.950414 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.950389    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b065d348-957d-4e03-b951-eccd1952ec83-istio-podinfo\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.950691 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.950674    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-data\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:08.974886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:08.974849    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"]
Nov 02 18:19:09.051732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051696    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b065d348-957d-4e03-b951-eccd1952ec83-istio-podinfo\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.051922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051750    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-data\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.051922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051805    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-socket\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.051922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051828    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-envoy\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.051922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051845    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b065d348-957d-4e03-b951-eccd1952ec83-istiod-ca-cert\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.051922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051889    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-credential-socket\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.051922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051921    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-istio-token\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.052315 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.051946    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-certs\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.052315 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.052015    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4mb\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-kube-api-access-mk4mb\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.052315 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.052277    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-data\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.052479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.052378    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-credential-socket\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.052595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.052506    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-certs\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.052743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.052712    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b065d348-957d-4e03-b951-eccd1952ec83-istiod-ca-cert\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.052897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.052880    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-socket\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.054816 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.054769    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-envoy\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.055099 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.055079    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b065d348-957d-4e03-b951-eccd1952ec83-istio-podinfo\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.109843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.109807    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-istio-token\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.109976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.109925    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4mb\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-kube-api-access-mk4mb\") pod \"e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") " pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.183186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.183074    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:09.622671 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.622638    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"]
Nov 02 18:19:09.623831 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:19:09.623801    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb065d348_957d_4e03_b951_eccd1952ec83.slice/crio-31a1756643c372fafd65c55796269cc0e924c904d7544035b75e89bf04135724 WatchSource:0}: Error finding container 31a1756643c372fafd65c55796269cc0e924c904d7544035b75e89bf04135724: Status 404 returned error can't find the container with id 31a1756643c372fafd65c55796269cc0e924c904d7544035b75e89bf04135724
Nov 02 18:19:09.626526 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.626497    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:19:09.626614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.626569    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:19:09.626614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.626598    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:19:09.936786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.936704    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h" event={"ID":"b065d348-957d-4e03-b951-eccd1952ec83","Type":"ContainerStarted","Data":"1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4"}
Nov 02 18:19:09.936786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:09.936757    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h" event={"ID":"b065d348-957d-4e03-b951-eccd1952ec83","Type":"ContainerStarted","Data":"31a1756643c372fafd65c55796269cc0e924c904d7544035b75e89bf04135724"}
Nov 02 18:19:10.054034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:10.053899    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h" podStartSLOduration=2.05387707 podStartE2EDuration="2.05387707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:10.052665536 +0000 UTC m=+3380.309696157" watchObservedRunningTime="2026-11-02 18:19:10.05387707 +0000 UTC m=+3380.310907691"
Nov 02 18:19:10.183738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:10.183705    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:10.189474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:10.189409    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:10.943295 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:10.943267    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:10.944347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:10.944327    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:19:29.053568 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:19:29.053528    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:19:30.968516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:30.968478    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"]
Nov 02 18:19:32.367316 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.367286    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"]
Nov 02 18:19:32.371728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.371710    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.430921 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.430886    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-tmp-dir\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.431103 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.430995    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-home\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.431103 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.431068    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-dshm\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.431250 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.431223    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-model-cache\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.431310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.431254    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860dec0-d876-4748-9b1e-d44c7649d9bf-tls-certs\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.431310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.431284    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nxv\" (UniqueName: \"kubernetes.io/projected/3860dec0-d876-4748-9b1e-d44c7649d9bf-kube-api-access-h4nxv\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.475768 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.475733    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-bcda4b\"/\"openshift-service-ca.crt\""
Nov 02 18:19:32.476034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.475733    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-bcda4b\"/\"default-dockercfg-r8p8l\""
Nov 02 18:19:32.476034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.475783    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-bcda4b\"/\"kube-root-ca.crt\""
Nov 02 18:19:32.476034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.475836    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-bcda4b\"/\"noten-model-bcda4b-kserve-self-signed-certs\""
Nov 02 18:19:32.476369 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.476345    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"]
Nov 02 18:19:32.532561 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.532529    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-model-cache\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.532561 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.532565    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860dec0-d876-4748-9b1e-d44c7649d9bf-tls-certs\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.532823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.532583    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nxv\" (UniqueName: \"kubernetes.io/projected/3860dec0-d876-4748-9b1e-d44c7649d9bf-kube-api-access-h4nxv\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.532823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.532715    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-tmp-dir\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.532823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.532746    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-home\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.532823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.532770    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-dshm\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.533039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.532961    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-model-cache\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.533039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.533027    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-tmp-dir\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.533210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.533185    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-home\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.535111 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.535090    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-dshm\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.535510 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.535491    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860dec0-d876-4748-9b1e-d44c7649d9bf-tls-certs\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.591239 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.591206    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nxv\" (UniqueName: \"kubernetes.io/projected/3860dec0-d876-4748-9b1e-d44c7649d9bf-kube-api-access-h4nxv\") pod \"noten-model-bcda4b-kserve-cfbc87cdf-2wj6j\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") " pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:32.684949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:32.684858    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:33.065770 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:33.065742    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"]
Nov 02 18:19:33.067849 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:19:33.067822    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3860dec0_d876_4748_9b1e_d44c7649d9bf.slice/crio-04137284fcf671b82abb302fa9fb32d5b17d32778fa2e589c96fff27bed4a211 WatchSource:0}: Error finding container 04137284fcf671b82abb302fa9fb32d5b17d32778fa2e589c96fff27bed4a211: Status 404 returned error can't find the container with id 04137284fcf671b82abb302fa9fb32d5b17d32778fa2e589c96fff27bed4a211
Nov 02 18:19:33.081329 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:33.081303    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j" event={"ID":"3860dec0-d876-4748-9b1e-d44c7649d9bf","Type":"ContainerStarted","Data":"04137284fcf671b82abb302fa9fb32d5b17d32778fa2e589c96fff27bed4a211"}
Nov 02 18:19:34.089227 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:34.089192    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j" event={"ID":"3860dec0-d876-4748-9b1e-d44c7649d9bf","Type":"ContainerStarted","Data":"7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43"}
Nov 02 18:19:34.207033 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:34.206977    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j" podStartSLOduration=2.206961701 podStartE2EDuration="2.206961701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:34.204820372 +0000 UTC m=+3404.461850995" watchObservedRunningTime="2026-11-02 18:19:34.206961701 +0000 UTC m=+3404.463992320"
Nov 02 18:19:34.685915 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:34.685877    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:42.685638 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:42.685598    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:42.698092 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:42.698070    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:43.155572 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:43.155539    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:19:45.990383 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:45.990342    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs" podUID="dd1cecd5-11db-4379-b6c4-289351da9614" containerName="main" containerID="cri-o://78083e0cd82c773b14f4198c467c4f354a938567c32a5c8183a446c1b2865b42" gracePeriod=45
Nov 02 18:19:46.171389 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.171358    4956 generic.go:356] "Generic (PLEG): container finished" podID="dd1cecd5-11db-4379-b6c4-289351da9614" containerID="78083e0cd82c773b14f4198c467c4f354a938567c32a5c8183a446c1b2865b42" exitCode=0
Nov 02 18:19:46.171558 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.171438    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs" event={"ID":"dd1cecd5-11db-4379-b6c4-289351da9614","Type":"ContainerDied","Data":"78083e0cd82c773b14f4198c467c4f354a938567c32a5c8183a446c1b2865b42"}
Nov 02 18:19:46.249823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.249755    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:19:46.294416 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294380    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd1cecd5-11db-4379-b6c4-289351da9614-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1cecd5-11db-4379-b6c4-289351da9614-tls-certs\") pod \"dd1cecd5-11db-4379-b6c4-289351da9614\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") "
Nov 02 18:19:46.294585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294454    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-home\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-home\") pod \"dd1cecd5-11db-4379-b6c4-289351da9614\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") "
Nov 02 18:19:46.294585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294546    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-tmp-dir\") pod \"dd1cecd5-11db-4379-b6c4-289351da9614\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") "
Nov 02 18:19:46.294680 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294595    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-model-cache\") pod \"dd1cecd5-11db-4379-b6c4-289351da9614\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") "
Nov 02 18:19:46.294742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294685    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-home" pod "dd1cecd5-11db-4379-b6c4-289351da9614" (UID: "dd1cecd5-11db-4379-b6c4-289351da9614"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:19:46.294742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294727    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1cecd5-11db-4379-b6c4-289351da9614-kube-api-access-jfmdn\" (UniqueName: \"kubernetes.io/projected/dd1cecd5-11db-4379-b6c4-289351da9614-kube-api-access-jfmdn\") pod \"dd1cecd5-11db-4379-b6c4-289351da9614\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") "
Nov 02 18:19:46.294857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294768    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-dshm\") pod \"dd1cecd5-11db-4379-b6c4-289351da9614\" (UID: \"dd1cecd5-11db-4379-b6c4-289351da9614\") "
Nov 02 18:19:46.294857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294806    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-tmp-dir" pod "dd1cecd5-11db-4379-b6c4-289351da9614" (UID: "dd1cecd5-11db-4379-b6c4-289351da9614"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:19:46.294959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.294899    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-model-cache" pod "dd1cecd5-11db-4379-b6c4-289351da9614" (UID: "dd1cecd5-11db-4379-b6c4-289351da9614"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:19:46.295277 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.295255    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:46.295277 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.295275    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:46.295612 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.295283    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:46.296794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.296766    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1cecd5-11db-4379-b6c4-289351da9614-tls-certs" pod "dd1cecd5-11db-4379-b6c4-289351da9614" (UID: "dd1cecd5-11db-4379-b6c4-289351da9614"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:46.296904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.296876    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-dshm" pod "dd1cecd5-11db-4379-b6c4-289351da9614" (UID: "dd1cecd5-11db-4379-b6c4-289351da9614"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:19:46.296950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.296926    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1cecd5-11db-4379-b6c4-289351da9614-kube-api-access-jfmdn" pod "dd1cecd5-11db-4379-b6c4-289351da9614" (UID: "dd1cecd5-11db-4379-b6c4-289351da9614"). InnerVolumeSpecName "kube-api-access-jfmdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:19:46.396817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.396768    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmdn\" (UniqueName: \"kubernetes.io/projected/dd1cecd5-11db-4379-b6c4-289351da9614-kube-api-access-jfmdn\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:46.396817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.396820    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1cecd5-11db-4379-b6c4-289351da9614-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:46.396817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:46.396831    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1cecd5-11db-4379-b6c4-289351da9614-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:19:47.179594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:47.179566    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"
Nov 02 18:19:47.179594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:47.179577    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs" event={"ID":"dd1cecd5-11db-4379-b6c4-289351da9614","Type":"ContainerDied","Data":"e2abd3b63906c8f9655fb2ef6d1c743845f9627262a47e70da3386d41aff6bfd"}
Nov 02 18:19:47.180190 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:47.179625    4956 scope.go:122] "RemoveContainer" containerID="78083e0cd82c773b14f4198c467c4f354a938567c32a5c8183a446c1b2865b42"
Nov 02 18:19:47.343302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:47.343264    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"]
Nov 02 18:19:47.347735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:47.347709    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-eb709f/autores-model-eb709f-kserve-69bf85555c-drwzs"]
Nov 02 18:19:47.865779 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:47.865743    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1cecd5-11db-4379-b6c4-289351da9614" path="/var/lib/kubelet/pods/dd1cecd5-11db-4379-b6c4-289351da9614/volumes"
Nov 02 18:19:49.894320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.894287    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"]
Nov 02 18:19:49.895031 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.895014    4956 cpu_manager.go:414] "containerMap: removing container" podUID="dd1cecd5-11db-4379-b6c4-289351da9614" containerName="main"
Nov 02 18:19:49.895092 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.895033    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1cecd5-11db-4379-b6c4-289351da9614" containerName="main"
Nov 02 18:19:49.895183 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.895173    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1cecd5-11db-4379-b6c4-289351da9614" containerName="main"
Nov 02 18:19:49.899785 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.899767    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:49.935757 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.935722    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:49.935924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.935858    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w862d\" (UniqueName: \"kubernetes.io/projected/b0a914c3-1d52-4610-a7cb-d3d50ec65664-kube-api-access-w862d\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:49.935971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.935951    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a914c3-1d52-4610-a7cb-d3d50ec65664-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:49.936050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:49.936035    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-metrics-certs\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.002848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.002808    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a9496c-serving-cert\""
Nov 02 18:19:50.005892 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.005866    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"]
Nov 02 18:19:50.037036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.036999    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.037241 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.037140    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w862d\" (UniqueName: \"kubernetes.io/projected/b0a914c3-1d52-4610-a7cb-d3d50ec65664-kube-api-access-w862d\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.037241 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:19:50.037174    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-serving-cert: secret "maas-api-e2e-extmod-a-a9496c-serving-cert" not found
Nov 02 18:19:50.037241 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.037203    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a914c3-1d52-4610-a7cb-d3d50ec65664-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.037241 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:19:50.037237    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls podName:b0a914c3-1d52-4610-a7cb-d3d50ec65664 nodeName:}" failed. No retries permitted until 2026-11-02 18:19:50.537219999 +0000 UTC m=+3420.794250624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls") pod "maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f" (UID: "b0a914c3-1d52-4610-a7cb-d3d50ec65664") : secret "maas-api-e2e-extmod-a-a9496c-serving-cert" not found
Nov 02 18:19:50.037461 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.037337    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-metrics-certs\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.037986 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.037963    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a914c3-1d52-4610-a7cb-d3d50ec65664-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.040575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.040551    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-metrics-certs\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.096933 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.096893    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w862d\" (UniqueName: \"kubernetes.io/projected/b0a914c3-1d52-4610-a7cb-d3d50ec65664-kube-api-access-w862d\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.544465 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.544433    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.547052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.547022    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:50.814136 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:50.814041    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:51.192529 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:51.192499    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"]
Nov 02 18:19:51.194231 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:19:51.194192    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a914c3_1d52_4610_a7cb_d3d50ec65664.slice/crio-0d2369dc970b88017a5cfd262c7803309ad4d60c80d7319655c7c9fb2240b3e4 WatchSource:0}: Error finding container 0d2369dc970b88017a5cfd262c7803309ad4d60c80d7319655c7c9fb2240b3e4: Status 404 returned error can't find the container with id 0d2369dc970b88017a5cfd262c7803309ad4d60c80d7319655c7c9fb2240b3e4
Nov 02 18:19:51.209626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:51.209598    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f" event={"ID":"b0a914c3-1d52-4610-a7cb-d3d50ec65664","Type":"ContainerStarted","Data":"0d2369dc970b88017a5cfd262c7803309ad4d60c80d7319655c7c9fb2240b3e4"}
Nov 02 18:19:52.219306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:52.219253    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f" event={"ID":"b0a914c3-1d52-4610-a7cb-d3d50ec65664","Type":"ContainerStarted","Data":"6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6"}
Nov 02 18:19:52.219752 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:52.219459    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:52.337565 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:52.337447    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f" podStartSLOduration=2.548071697 podStartE2EDuration="3.33742915s" totalImagesPullingTime="789.357453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:52.335698931 +0000 UTC m=+3422.592729551" watchObservedRunningTime="2026-11-02 18:19:52.33742915 +0000 UTC m=+3422.594459770"
Nov 02 18:19:58.232976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:19:58.232944    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:19:59.054114 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:19:59.054072    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:20:11.684984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:11.684946    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"]
Nov 02 18:20:11.849744 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:11.849708    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"]
Nov 02 18:20:11.849947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:11.849919    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h" podUID="b065d348-957d-4e03-b951-eccd1952ec83" containerName="istio-proxy" containerID="cri-o://1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4" gracePeriod=30
Nov 02 18:20:13.940564 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:13.940524    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"]
Nov 02 18:20:17.136351 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.136327    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:20:17.167826 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.167794    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-socket\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.167856    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-kube-api-access-mk4mb\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-kube-api-access-mk4mb\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.167893    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-credential-socket\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.167951    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-envoy\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168094    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-istio-token\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-istio-token\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168151    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-socket" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:17.168182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168167    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-certs\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168206    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-data\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168267    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b065d348-957d-4e03-b951-eccd1952ec83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b065d348-957d-4e03-b951-eccd1952ec83-istio-podinfo\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168309    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b065d348-957d-4e03-b951-eccd1952ec83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b065d348-957d-4e03-b951-eccd1952ec83-istiod-ca-cert\") pod \"b065d348-957d-4e03-b951-eccd1952ec83\" (UID: \"b065d348-957d-4e03-b951-eccd1952ec83\") "
Nov 02 18:20:17.168343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168307    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-credential-socket" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:17.168577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168441    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-certs" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:17.168754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168697    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b065d348-957d-4e03-b951-eccd1952ec83-istiod-ca-cert" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:17.168915 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168885    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-data" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:17.168915 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168826    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.169032 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168914    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b065d348-957d-4e03-b951-eccd1952ec83-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.169032 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168926    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.169032 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.168937    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.170618 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.170590    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-kube-api-access-mk4mb" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "kube-api-access-mk4mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:17.170745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.170632    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-envoy" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:17.170913 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.170896    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-istio-token" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:17.171060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.171041    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b065d348-957d-4e03-b951-eccd1952ec83-istio-podinfo" pod "b065d348-957d-4e03-b951-eccd1952ec83" (UID: "b065d348-957d-4e03-b951-eccd1952ec83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:20:17.269913 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.269879    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b065d348-957d-4e03-b951-eccd1952ec83-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.269913 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.269909    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4mb\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-kube-api-access-mk4mb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.269913 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.269920    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.270184 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.269928    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b065d348-957d-4e03-b951-eccd1952ec83-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.270184 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.269936    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b065d348-957d-4e03-b951-eccd1952ec83-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:17.389558 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.389472    4956 generic.go:356] "Generic (PLEG): container finished" podID="b065d348-957d-4e03-b951-eccd1952ec83" containerID="1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4" exitCode=0
Nov 02 18:20:17.389558 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.389525    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"
Nov 02 18:20:17.389738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.389560    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h" event={"ID":"b065d348-957d-4e03-b951-eccd1952ec83","Type":"ContainerDied","Data":"1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4"}
Nov 02 18:20:17.389738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.389613    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h" event={"ID":"b065d348-957d-4e03-b951-eccd1952ec83","Type":"ContainerDied","Data":"31a1756643c372fafd65c55796269cc0e924c904d7544035b75e89bf04135724"}
Nov 02 18:20:17.389738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.389638    4956 scope.go:122] "RemoveContainer" containerID="1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4"
Nov 02 18:20:17.400848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.400828    4956 scope.go:122] "RemoveContainer" containerID="1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4"
Nov 02 18:20:17.401182 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:20:17.401162    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4\": container with ID starting with 1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4 not found: ID does not exist" containerID="1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4"
Nov 02 18:20:17.401231 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.401194    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4"} err="failed to get container status \"1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4\": rpc error: code = NotFound desc = could not find container \"1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4\": container with ID starting with 1bc5f96bed23e202af7b8805ed4c7eb132ea177c4c7dc54c3c55fab7160b47b4 not found: ID does not exist"
Nov 02 18:20:17.557781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.557732    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"]
Nov 02 18:20:17.570421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.563349    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-bcda4b-openshift-default-7f86746597-fhj7h"]
Nov 02 18:20:17.865886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:17.865852    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b065d348-957d-4e03-b951-eccd1952ec83" path="/var/lib/kubelet/pods/b065d348-957d-4e03-b951-eccd1952ec83/volumes"
Nov 02 18:20:18.941552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:18.941507    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866" podUID="5ecdd73b-975c-4145-8da6-7f9edd9d8e61" containerName="payload-processing" containerID="cri-o://7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb" gracePeriod=25
Nov 02 18:20:19.208515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.208483    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:20:19.291657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.291622    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-kube-api-access-w2pp7\" (UniqueName: \"kubernetes.io/projected/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-kube-api-access-w2pp7\") pod \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") "
Nov 02 18:20:19.291843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.291774    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-plugins-config-volume\") pod \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\" (UID: \"5ecdd73b-975c-4145-8da6-7f9edd9d8e61\") "
Nov 02 18:20:19.292192 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.292155    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-plugins-config-volume" pod "5ecdd73b-975c-4145-8da6-7f9edd9d8e61" (UID: "5ecdd73b-975c-4145-8da6-7f9edd9d8e61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:19.293875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.293849    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-kube-api-access-w2pp7" pod "5ecdd73b-975c-4145-8da6-7f9edd9d8e61" (UID: "5ecdd73b-975c-4145-8da6-7f9edd9d8e61"). InnerVolumeSpecName "kube-api-access-w2pp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:19.393605 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.393568    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:19.393605 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.393599    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2pp7\" (UniqueName: \"kubernetes.io/projected/5ecdd73b-975c-4145-8da6-7f9edd9d8e61-kube-api-access-w2pp7\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:19.403802 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.403766    4956 generic.go:356] "Generic (PLEG): container finished" podID="5ecdd73b-975c-4145-8da6-7f9edd9d8e61" containerID="7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb" exitCode=0
Nov 02 18:20:19.403966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.403812    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"
Nov 02 18:20:19.403966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.403858    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866" event={"ID":"5ecdd73b-975c-4145-8da6-7f9edd9d8e61","Type":"ContainerDied","Data":"7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb"}
Nov 02 18:20:19.403966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.403912    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866" event={"ID":"5ecdd73b-975c-4145-8da6-7f9edd9d8e61","Type":"ContainerDied","Data":"345ae1db70e5411e46161dc065abd95a2ffe3f4ce69d5d58e1c73b2eb5932e6f"}
Nov 02 18:20:19.403966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.403933    4956 scope.go:122] "RemoveContainer" containerID="7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb"
Nov 02 18:20:19.414700 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.414678    4956 scope.go:122] "RemoveContainer" containerID="7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb"
Nov 02 18:20:19.414996 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:20:19.414978    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb\": container with ID starting with 7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb not found: ID does not exist" containerID="7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb"
Nov 02 18:20:19.415047 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.415005    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb"} err="failed to get container status \"7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb\": rpc error: code = NotFound desc = could not find container \"7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb\": container with ID starting with 7cf973dd4fa2ad0b4d104924a94ae22e244956ed422f6c34c389f7e11c8cedeb not found: ID does not exist"
Nov 02 18:20:19.568003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.567912    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"]
Nov 02 18:20:19.571260 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.571223    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eb709f-cc5ddb4d7-vm866"]
Nov 02 18:20:19.865436 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:19.865355    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecdd73b-975c-4145-8da6-7f9edd9d8e61" path="/var/lib/kubelet/pods/5ecdd73b-975c-4145-8da6-7f9edd9d8e61/volumes"
Nov 02 18:20:26.706152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:26.706079    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j" podUID="3860dec0-d876-4748-9b1e-d44c7649d9bf" containerName="main" containerID="cri-o://7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43" gracePeriod=45
Nov 02 18:20:26.992813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:26.992784    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:20:27.089085 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089050    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-model-cache\") pod \"3860dec0-d876-4748-9b1e-d44c7649d9bf\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") "
Nov 02 18:20:27.089319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089141    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-home\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-home\") pod \"3860dec0-d876-4748-9b1e-d44c7649d9bf\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") "
Nov 02 18:20:27.089319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089221    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-dshm\") pod \"3860dec0-d876-4748-9b1e-d44c7649d9bf\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") "
Nov 02 18:20:27.089319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089253    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3860dec0-d876-4748-9b1e-d44c7649d9bf-kube-api-access-h4nxv\" (UniqueName: \"kubernetes.io/projected/3860dec0-d876-4748-9b1e-d44c7649d9bf-kube-api-access-h4nxv\") pod \"3860dec0-d876-4748-9b1e-d44c7649d9bf\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") "
Nov 02 18:20:27.089484 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089342    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3860dec0-d876-4748-9b1e-d44c7649d9bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/3860dec0-d876-4748-9b1e-d44c7649d9bf-tls-certs\") pod \"3860dec0-d876-4748-9b1e-d44c7649d9bf\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") "
Nov 02 18:20:27.089484 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089368    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-model-cache" pod "3860dec0-d876-4748-9b1e-d44c7649d9bf" (UID: "3860dec0-d876-4748-9b1e-d44c7649d9bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:27.089484 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089415    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-home" pod "3860dec0-d876-4748-9b1e-d44c7649d9bf" (UID: "3860dec0-d876-4748-9b1e-d44c7649d9bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:27.089484 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089449    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-tmp-dir\") pod \"3860dec0-d876-4748-9b1e-d44c7649d9bf\" (UID: \"3860dec0-d876-4748-9b1e-d44c7649d9bf\") "
Nov 02 18:20:27.089746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089719    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-tmp-dir" pod "3860dec0-d876-4748-9b1e-d44c7649d9bf" (UID: "3860dec0-d876-4748-9b1e-d44c7649d9bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:27.090052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.089988    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:27.090318 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.090095    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:27.090318 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.090208    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:27.091702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.091672    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3860dec0-d876-4748-9b1e-d44c7649d9bf-tls-certs" pod "3860dec0-d876-4748-9b1e-d44c7649d9bf" (UID: "3860dec0-d876-4748-9b1e-d44c7649d9bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:27.091797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.091721    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-dshm" pod "3860dec0-d876-4748-9b1e-d44c7649d9bf" (UID: "3860dec0-d876-4748-9b1e-d44c7649d9bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:20:27.091797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.091777    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3860dec0-d876-4748-9b1e-d44c7649d9bf-kube-api-access-h4nxv" pod "3860dec0-d876-4748-9b1e-d44c7649d9bf" (UID: "3860dec0-d876-4748-9b1e-d44c7649d9bf"). InnerVolumeSpecName "kube-api-access-h4nxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:27.190996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.190962    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860dec0-d876-4748-9b1e-d44c7649d9bf-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:27.190996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.190989    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nxv\" (UniqueName: \"kubernetes.io/projected/3860dec0-d876-4748-9b1e-d44c7649d9bf-kube-api-access-h4nxv\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:27.190996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.191002    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860dec0-d876-4748-9b1e-d44c7649d9bf-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:20:27.455182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.455149    4956 generic.go:356] "Generic (PLEG): container finished" podID="3860dec0-d876-4748-9b1e-d44c7649d9bf" containerID="7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43" exitCode=0
Nov 02 18:20:27.455335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.455203    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j" event={"ID":"3860dec0-d876-4748-9b1e-d44c7649d9bf","Type":"ContainerDied","Data":"7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43"}
Nov 02 18:20:27.455335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.455237    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j" event={"ID":"3860dec0-d876-4748-9b1e-d44c7649d9bf","Type":"ContainerDied","Data":"04137284fcf671b82abb302fa9fb32d5b17d32778fa2e589c96fff27bed4a211"}
Nov 02 18:20:27.455335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.455255    4956 scope.go:122] "RemoveContainer" containerID="7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43"
Nov 02 18:20:27.455335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.455204    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"
Nov 02 18:20:27.466278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.466256    4956 scope.go:122] "RemoveContainer" containerID="7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43"
Nov 02 18:20:27.466575 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:20:27.466554    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43\": container with ID starting with 7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43 not found: ID does not exist" containerID="7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43"
Nov 02 18:20:27.466622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.466585    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43"} err="failed to get container status \"7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43\": rpc error: code = NotFound desc = could not find container \"7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43\": container with ID starting with 7fdde4a431123233f8e3c328d4278b22c2a96712644c81d1899fc9baf9d69a43 not found: ID does not exist"
Nov 02 18:20:27.629405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.629364    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"]
Nov 02 18:20:27.678691 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.678653    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-bcda4b/noten-model-bcda4b-kserve-cfbc87cdf-2wj6j"]
Nov 02 18:20:27.866650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:27.866566    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3860dec0-d876-4748-9b1e-d44c7649d9bf" path="/var/lib/kubelet/pods/3860dec0-d876-4748-9b1e-d44c7649d9bf/volumes"
Nov 02 18:20:29.053353 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:20:29.053321    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:20:56.288772 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.288738    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"]
Nov 02 18:20:56.289504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289473    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3860dec0-d876-4748-9b1e-d44c7649d9bf" containerName="main"
Nov 02 18:20:56.289504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289491    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3860dec0-d876-4748-9b1e-d44c7649d9bf" containerName="main"
Nov 02 18:20:56.289504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289511    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5ecdd73b-975c-4145-8da6-7f9edd9d8e61" containerName="payload-processing"
Nov 02 18:20:56.289749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289518    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecdd73b-975c-4145-8da6-7f9edd9d8e61" containerName="payload-processing"
Nov 02 18:20:56.289749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289529    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b065d348-957d-4e03-b951-eccd1952ec83" containerName="istio-proxy"
Nov 02 18:20:56.289749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289534    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b065d348-957d-4e03-b951-eccd1952ec83" containerName="istio-proxy"
Nov 02 18:20:56.289749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289651    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3860dec0-d876-4748-9b1e-d44c7649d9bf" containerName="main"
Nov 02 18:20:56.289749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289665    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecdd73b-975c-4145-8da6-7f9edd9d8e61" containerName="payload-processing"
Nov 02 18:20:56.289749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.289673    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b065d348-957d-4e03-b951-eccd1952ec83" containerName="istio-proxy"
Nov 02 18:20:56.293272 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.293255    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.347425 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.347388    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-508b8f-openshift-default-dockercfg-r9bvl\""
Nov 02 18:20:56.400169 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.400136    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"]
Nov 02 18:20:56.435895 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.435862    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-data\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.435923    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-podinfo\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.436032    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-credential-socket\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.436071    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-socket\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436276 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.436225    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-token\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.436327    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bea785-b1bf-45cd-bd9d-63678fb98a07-istiod-ca-cert\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.436370    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-certs\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436476 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.436455    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-envoy\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.436517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.436508    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnhg\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-kube-api-access-4xnhg\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.537617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537574    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-data\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.537833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537634    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-podinfo\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.537833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537682    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-credential-socket\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.537833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537716    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-socket\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.537833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537791    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-token\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537875    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bea785-b1bf-45cd-bd9d-63678fb98a07-istiod-ca-cert\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537909    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-certs\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.537949    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-envoy\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.538010    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnhg\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-kube-api-access-4xnhg\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538275 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.538052    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-data\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538275 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.538087    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-socket\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538275 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.538137    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-credential-socket\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538419 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.538362    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-certs\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.538935 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.538864    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bea785-b1bf-45cd-bd9d-63678fb98a07-istiod-ca-cert\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.540381 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.540360    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-envoy\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.540565 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.540544    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-podinfo\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.595105 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.595061    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-token\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.595280 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.595200    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnhg\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-kube-api-access-4xnhg\") pod \"e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") " pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:56.607108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:56.607067    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:57.044433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:57.044396    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"]
Nov 02 18:20:57.083747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:57.083714    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:20:57.083848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:57.083783    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:20:57.083848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:57.083812    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:20:57.643428 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:57.643388    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" event={"ID":"10bea785-b1bf-45cd-bd9d-63678fb98a07","Type":"ContainerStarted","Data":"fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b"}
Nov 02 18:20:57.643926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:57.643442    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" event={"ID":"10bea785-b1bf-45cd-bd9d-63678fb98a07","Type":"ContainerStarted","Data":"ccc509a3e7b125a53380772f5c3835233fbb1d4383a0667866382d135bc68ab7"}
Nov 02 18:20:57.761561 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:57.761511    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podStartSLOduration=1.7614972949999999 podStartE2EDuration="1.761497295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:57.759732795 +0000 UTC m=+3488.016763414" watchObservedRunningTime="2026-11-02 18:20:57.761497295 +0000 UTC m=+3488.018527915"
Nov 02 18:20:58.607394 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:58.607352    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:58.613524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:58.613497    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:58.650082 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:58.650051    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:58.651198 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:20:58.651172    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:20:59.057536 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:20:59.057494    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:21:01.606001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.605968    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"]
Nov 02 18:21:01.613894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.613863    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.689784 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.689747    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"]
Nov 02 18:21:01.704494 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704449    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704676 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704574    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704676 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704642    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704713    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704835 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704806    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv82\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-kube-api-access-xnv82\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704835 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704831    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704940 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704926    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704945    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.704992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.704973    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.718929 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.718900    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e4b90c-openshift-default-dockercfg-6kx9n\""
Nov 02 18:21:01.805946 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.805910    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.805946 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.805954    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806273 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.805998    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806273 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806073    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806273 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806147    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806273 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806219    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806273 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806251    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806317    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv82\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-kube-api-access-xnv82\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806356    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806501    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806533    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806757 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806567    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806857    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.806886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.806881    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.808834 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.808810    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.808950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.808842    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.864029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.863938    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.864662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.864638    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv82\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-kube-api-access-xnv82\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:01.929540 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:01.929502    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:02.359233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.359203    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"]
Nov 02 18:21:02.359977 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:21:02.359953    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7d4c4c_2363_4cd5_903b_10832fe15b8a.slice/crio-8875d8e6bf27325d59b7889045e0c44437bf66793e69ac76f737ea2cce360f2a WatchSource:0}: Error finding container 8875d8e6bf27325d59b7889045e0c44437bf66793e69ac76f737ea2cce360f2a: Status 404 returned error can't find the container with id 8875d8e6bf27325d59b7889045e0c44437bf66793e69ac76f737ea2cce360f2a
Nov 02 18:21:02.362822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.362790    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:02.362905 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.362855    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:02.362905 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.362885    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:02.676201 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.676080    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" event={"ID":"3f7d4c4c-2363-4cd5-903b-10832fe15b8a","Type":"ContainerStarted","Data":"1fab816052b0b3334280ac04544fb3de250d859b353d5595f39d2e4983bd0354"}
Nov 02 18:21:02.676201 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.676157    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" event={"ID":"3f7d4c4c-2363-4cd5-903b-10832fe15b8a","Type":"ContainerStarted","Data":"8875d8e6bf27325d59b7889045e0c44437bf66793e69ac76f737ea2cce360f2a"}
Nov 02 18:21:02.799390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.799326    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" podStartSLOduration=1.799304597 podStartE2EDuration="1.799304597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:02.794142191 +0000 UTC m=+3493.051172808" watchObservedRunningTime="2026-11-02 18:21:02.799304597 +0000 UTC m=+3493.056335217"
Nov 02 18:21:02.930066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:02.929975    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:03.935716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:03.935686    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:04.518144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.518090    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"]
Nov 02 18:21:04.522326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.522307    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:04.575930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.575893    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-508b8f\""
Nov 02 18:21:04.576158 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.576027    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-508b8f-dockercfg-xwwhm\""
Nov 02 18:21:04.579611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.579584    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"]
Nov 02 18:21:04.653600 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.653551    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4c0f49-197e-44c8-ac7f-016c175044ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:04.654057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.654026    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zbm\" (UniqueName: \"kubernetes.io/projected/1b4c0f49-197e-44c8-ac7f-016c175044ff-kube-api-access-m2zbm\") pod \"payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:04.690009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.689982    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:04.690922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.690903    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:21:04.755412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.755372    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zbm\" (UniqueName: \"kubernetes.io/projected/1b4c0f49-197e-44c8-ac7f-016c175044ff-kube-api-access-m2zbm\") pod \"payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:04.755612 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.755499    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4c0f49-197e-44c8-ac7f-016c175044ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:04.756221 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.756201    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4c0f49-197e-44c8-ac7f-016c175044ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:04.819206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.819100    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zbm\" (UniqueName: \"kubernetes.io/projected/1b4c0f49-197e-44c8-ac7f-016c175044ff-kube-api-access-m2zbm\") pod \"payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:04.837812 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:04.837774    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:05.232751 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:05.232720    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"]
Nov 02 18:21:05.235113 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:21:05.235085    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4c0f49_197e_44c8_ac7f_016c175044ff.slice/crio-1ed0f0c2246f0bbf7e81787fa3f15d7bf9be6540f775c242198c6396cd9ef658 WatchSource:0}: Error finding container 1ed0f0c2246f0bbf7e81787fa3f15d7bf9be6540f775c242198c6396cd9ef658: Status 404 returned error can't find the container with id 1ed0f0c2246f0bbf7e81787fa3f15d7bf9be6540f775c242198c6396cd9ef658
Nov 02 18:21:05.697329 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:05.697227    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd" event={"ID":"1b4c0f49-197e-44c8-ac7f-016c175044ff","Type":"ContainerStarted","Data":"aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130"}
Nov 02 18:21:05.697329 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:05.697285    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd" event={"ID":"1b4c0f49-197e-44c8-ac7f-016c175044ff","Type":"ContainerStarted","Data":"1ed0f0c2246f0bbf7e81787fa3f15d7bf9be6540f775c242198c6396cd9ef658"}
Nov 02 18:21:05.831283 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:05.831231    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd" podStartSLOduration=1.831214254 podStartE2EDuration="1.831214254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:05.828607716 +0000 UTC m=+3496.085638336" watchObservedRunningTime="2026-11-02 18:21:05.831214254 +0000 UTC m=+3496.088244873"
Nov 02 18:21:06.703628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:06.703590    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:11.981286 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:11.981247    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"]
Nov 02 18:21:11.985652 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:11.985631    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.039660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.039619    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e4b90c\""
Nov 02 18:21:12.064303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.064269    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jhgg\" (UniqueName: \"kubernetes.io/projected/4bb23074-96da-4827-b1fb-9f344e15b68a-kube-api-access-2jhgg\") pod \"payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.064480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.064324    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb23074-96da-4827-b1fb-9f344e15b68a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.089147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.089078    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e4b90c-dockercfg-p7sfg\""
Nov 02 18:21:12.092596 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.092572    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"]
Nov 02 18:21:12.165208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.165165    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jhgg\" (UniqueName: \"kubernetes.io/projected/4bb23074-96da-4827-b1fb-9f344e15b68a-kube-api-access-2jhgg\") pod \"payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.165384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.165234    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb23074-96da-4827-b1fb-9f344e15b68a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.166229 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.166207    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb23074-96da-4827-b1fb-9f344e15b68a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.222039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.222000    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jhgg\" (UniqueName: \"kubernetes.io/projected/4bb23074-96da-4827-b1fb-9f344e15b68a-kube-api-access-2jhgg\") pod \"payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.299561 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.299451    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:12.679210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.679178    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"]
Nov 02 18:21:12.680282 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:21:12.680259    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb23074_96da_4827_b1fb_9f344e15b68a.slice/crio-bdc480f98d91dd4d9024e25a9603481186d5d515fe529641cc7b5aaca0341e72 WatchSource:0}: Error finding container bdc480f98d91dd4d9024e25a9603481186d5d515fe529641cc7b5aaca0341e72: Status 404 returned error can't find the container with id bdc480f98d91dd4d9024e25a9603481186d5d515fe529641cc7b5aaca0341e72
Nov 02 18:21:12.748375 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:12.748332    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4" event={"ID":"4bb23074-96da-4827-b1fb-9f344e15b68a","Type":"ContainerStarted","Data":"bdc480f98d91dd4d9024e25a9603481186d5d515fe529641cc7b5aaca0341e72"}
Nov 02 18:21:13.756672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:13.756638    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4" event={"ID":"4bb23074-96da-4827-b1fb-9f344e15b68a","Type":"ContainerStarted","Data":"90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be"}
Nov 02 18:21:13.757084 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:13.756843    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:13.874511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:13.874452    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4" podStartSLOduration=2.874437104 podStartE2EDuration="2.874437104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:13.872274382 +0000 UTC m=+3504.129305000" watchObservedRunningTime="2026-11-02 18:21:13.874437104 +0000 UTC m=+3504.131467724"
Nov 02 18:21:17.713517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:17.713486    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:21:24.766465 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:24.766435    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:21:29.055489 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:21:29.055440    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:21:40.119895 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.119858    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"]
Nov 02 18:21:40.125416 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.125394    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.180571 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.180540    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"]
Nov 02 18:21:40.211758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.211713    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssbr\" (UniqueName: \"kubernetes.io/projected/202a923e-4e0c-42c0-8680-b60162fe8e5d-kube-api-access-vssbr\") pod \"payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.211982 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.211953    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a923e-4e0c-42c0-8680-b60162fe8e5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.226477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.226442    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a72f55\""
Nov 02 18:21:40.226477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.226473    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a72f55-dockercfg-p2cz8\""
Nov 02 18:21:40.313545 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.313506    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a923e-4e0c-42c0-8680-b60162fe8e5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.313746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.313613    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vssbr\" (UniqueName: \"kubernetes.io/projected/202a923e-4e0c-42c0-8680-b60162fe8e5d-kube-api-access-vssbr\") pod \"payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.314290 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.314268    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a923e-4e0c-42c0-8680-b60162fe8e5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.372026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.371949    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssbr\" (UniqueName: \"kubernetes.io/projected/202a923e-4e0c-42c0-8680-b60162fe8e5d-kube-api-access-vssbr\") pod \"payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.440183 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.440140    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.827446 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.827415    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"]
Nov 02 18:21:40.828873 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:21:40.828843    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202a923e_4e0c_42c0_8680_b60162fe8e5d.slice/crio-7b1fd296b2a53e05a21c8f19e3ea9fa84699babb4511b2f08f9db9c301c968a0 WatchSource:0}: Error finding container 7b1fd296b2a53e05a21c8f19e3ea9fa84699babb4511b2f08f9db9c301c968a0: Status 404 returned error can't find the container with id 7b1fd296b2a53e05a21c8f19e3ea9fa84699babb4511b2f08f9db9c301c968a0
Nov 02 18:21:40.935350 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.935315    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p" event={"ID":"202a923e-4e0c-42c0-8680-b60162fe8e5d","Type":"ContainerStarted","Data":"835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b"}
Nov 02 18:21:40.935511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.935382    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:40.935511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:40.935402    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p" event={"ID":"202a923e-4e0c-42c0-8680-b60162fe8e5d","Type":"ContainerStarted","Data":"7b1fd296b2a53e05a21c8f19e3ea9fa84699babb4511b2f08f9db9c301c968a0"}
Nov 02 18:21:41.053575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:41.053520    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p" podStartSLOduration=1.053503408 podStartE2EDuration="1.053503408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:41.051832528 +0000 UTC m=+3531.308863150" watchObservedRunningTime="2026-11-02 18:21:41.053503408 +0000 UTC m=+3531.310534027"
Nov 02 18:21:47.492164 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.492109    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"]
Nov 02 18:21:47.497830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.497784    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.553767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.553734    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-cd9ae6\""
Nov 02 18:21:47.553767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.553734    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-cd9ae6-dockercfg-b76gc\""
Nov 02 18:21:47.602824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.602776    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwp9\" (UniqueName: \"kubernetes.io/projected/99633e47-b96f-48cf-8cbb-5ccaba672c9b-kube-api-access-mxwp9\") pod \"payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.603018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.602915    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99633e47-b96f-48cf-8cbb-5ccaba672c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.605901 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.605871    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"]
Nov 02 18:21:47.633104 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.633068    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"]
Nov 02 18:21:47.637503 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.637478    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:47.697609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.697568    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"]
Nov 02 18:21:47.704241 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.704047    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwp9\" (UniqueName: \"kubernetes.io/projected/99633e47-b96f-48cf-8cbb-5ccaba672c9b-kube-api-access-mxwp9\") pod \"payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.704241 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.704174    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99633e47-b96f-48cf-8cbb-5ccaba672c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.705097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.705073    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99633e47-b96f-48cf-8cbb-5ccaba672c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.764875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.764794    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwp9\" (UniqueName: \"kubernetes.io/projected/99633e47-b96f-48cf-8cbb-5ccaba672c9b-kube-api-access-mxwp9\") pod \"payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.805341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.805306    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvd8\" (UniqueName: \"kubernetes.io/projected/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-kube-api-access-tzvd8\") pod \"payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:47.805504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.805371    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:47.812070 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.812045    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:47.906290 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.906252    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvd8\" (UniqueName: \"kubernetes.io/projected/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-kube-api-access-tzvd8\") pod \"payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:47.906482 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.906362    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:47.907171 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.907147    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:47.963533 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:47.963498    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvd8\" (UniqueName: \"kubernetes.io/projected/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-kube-api-access-tzvd8\") pod \"payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:48.192377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.192341    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"]
Nov 02 18:21:48.193438 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:21:48.193409    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99633e47_b96f_48cf_8cbb_5ccaba672c9b.slice/crio-88de6c18712f04cc39124e61e13e7cc382cb8b6d78ade6e276e98a0ab22af8a2 WatchSource:0}: Error finding container 88de6c18712f04cc39124e61e13e7cc382cb8b6d78ade6e276e98a0ab22af8a2: Status 404 returned error can't find the container with id 88de6c18712f04cc39124e61e13e7cc382cb8b6d78ade6e276e98a0ab22af8a2
Nov 02 18:21:48.258371 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.258152    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:48.646243 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.646209    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"]
Nov 02 18:21:48.646623 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:21:48.646252    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6546d7de_6ff3_4a57_a51c_d13eb7760e7d.slice/crio-72fab573274114feb64da3e4d93de5666f71ddb963a166714a3991b217632ace WatchSource:0}: Error finding container 72fab573274114feb64da3e4d93de5666f71ddb963a166714a3991b217632ace: Status 404 returned error can't find the container with id 72fab573274114feb64da3e4d93de5666f71ddb963a166714a3991b217632ace
Nov 02 18:21:48.987263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.987217    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565" event={"ID":"6546d7de-6ff3-4a57-a51c-d13eb7760e7d","Type":"ContainerStarted","Data":"d56def61b2d21a24a58fa849a87fb517022ed172037a79b25db56a0bdbeb83b4"}
Nov 02 18:21:48.987474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.987292    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:48.987474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.987315    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565" event={"ID":"6546d7de-6ff3-4a57-a51c-d13eb7760e7d","Type":"ContainerStarted","Data":"72fab573274114feb64da3e4d93de5666f71ddb963a166714a3991b217632ace"}
Nov 02 18:21:48.988866 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.988830    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8" event={"ID":"99633e47-b96f-48cf-8cbb-5ccaba672c9b","Type":"ContainerStarted","Data":"ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956"}
Nov 02 18:21:48.988990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.988884    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8" event={"ID":"99633e47-b96f-48cf-8cbb-5ccaba672c9b","Type":"ContainerStarted","Data":"88de6c18712f04cc39124e61e13e7cc382cb8b6d78ade6e276e98a0ab22af8a2"}
Nov 02 18:21:48.988990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:48.988908    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:21:49.105624 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:49.105566    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565" podStartSLOduration=2.105551896 podStartE2EDuration="2.105551896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:49.102531257 +0000 UTC m=+3539.359561879" watchObservedRunningTime="2026-11-02 18:21:49.105551896 +0000 UTC m=+3539.362582515"
Nov 02 18:21:49.216290 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:49.216168    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8" podStartSLOduration=2.216110544 podStartE2EDuration="2.216110544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:49.215647204 +0000 UTC m=+3539.472677840" watchObservedRunningTime="2026-11-02 18:21:49.216110544 +0000 UTC m=+3539.473141161"
Nov 02 18:21:51.944270 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:51.944238    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:21:59.053136 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:21:59.053084    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:21:59.999346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:59.999272    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:21:59.999346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:21:59.999339    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:22:29.053467 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:22:29.053429    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:22:50.376551 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:50.376523    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:22:50.378928 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:50.378101    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:22:55.425178 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:55.425056    4956 generic.go:356] "Generic (PLEG): container finished" podID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerID="fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b" exitCode=0
Nov 02 18:22:55.425601 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:55.425170    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" event={"ID":"10bea785-b1bf-45cd-bd9d-63678fb98a07","Type":"ContainerDied","Data":"fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b"}
Nov 02 18:22:55.425642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:55.425627    4956 scope.go:122] "RemoveContainer" containerID="fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b"
Nov 02 18:22:55.427366 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:55.427338    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:55.427450 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:55.427415    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:55.427491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:55.427448    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:56.460284 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:56.460251    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" event={"ID":"10bea785-b1bf-45cd-bd9d-63678fb98a07","Type":"ContainerStarted","Data":"5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628"}
Nov 02 18:22:56.608007 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:56.607971    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:22:56.608900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:56.608871    4956 patch_prober.go:29] interesting pod/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:22:56.609022 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:56.608921    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 18:22:57.607723 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:57.607681    4956 patch_prober.go:29] interesting pod/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:22:57.608258 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:57.607754    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 18:22:58.608467 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:58.608430    4956 patch_prober.go:29] interesting pod/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:22:58.608887 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:58.608484    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 18:22:58.650291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:58.650248    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:22:59.077440 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:22:59.077399    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:22:59.608099 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:59.608062    4956 patch_prober.go:29] interesting pod/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:22:59.608306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:22:59.608137    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 18:23:00.608242 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:00.608202    4956 patch_prober.go:29] interesting pod/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:23:00.608664 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:00.608276    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 18:23:01.607546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:01.607510    4956 patch_prober.go:29] interesting pod/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:23:01.607743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:01.607566    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 18:23:02.612891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:02.612859    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:23:02.614160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:02.614108    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:23:21.090034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.090003    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"]
Nov 02 18:23:21.095411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.095379    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.151680 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.151649    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"]
Nov 02 18:23:21.153266 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.153236    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-cd9ae6-serving-cert\""
Nov 02 18:23:21.192251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.192212    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8p92\" (UniqueName: \"kubernetes.io/projected/4d7310ab-a339-437a-a5ed-9daecd1a882a-kube-api-access-b8p92\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.192251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.192263    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-metrics-certs\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.192605 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.192578    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.192698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.192684    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d7310ab-a339-437a-a5ed-9daecd1a882a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.294279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.294245    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d7310ab-a339-437a-a5ed-9daecd1a882a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.294279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.294284    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8p92\" (UniqueName: \"kubernetes.io/projected/4d7310ab-a339-437a-a5ed-9daecd1a882a-kube-api-access-b8p92\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.294557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.294302    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-metrics-certs\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.294557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.294435    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.294557 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:23:21.294548    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-serving-cert: secret "maas-api-e2e-extmod-b-cd9ae6-serving-cert" not found
Nov 02 18:23:21.294733 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:23:21.294595    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls podName:4d7310ab-a339-437a-a5ed-9daecd1a882a nodeName:}" failed. No retries permitted until 2026-11-02 18:23:21.794581494 +0000 UTC m=+3632.051612093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls") pod "maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j" (UID: "4d7310ab-a339-437a-a5ed-9daecd1a882a") : secret "maas-api-e2e-extmod-b-cd9ae6-serving-cert" not found
Nov 02 18:23:21.294967 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.294941    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d7310ab-a339-437a-a5ed-9daecd1a882a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.297004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.296978    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-metrics-certs\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.355386 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.355301    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8p92\" (UniqueName: \"kubernetes.io/projected/4d7310ab-a339-437a-a5ed-9daecd1a882a-kube-api-access-b8p92\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.801875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.801834    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:21.804578 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:21.804552    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:22.011562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:22.011526    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:22.387735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:22.387707    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"]
Nov 02 18:23:22.388667 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:23:22.388637    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7310ab_a339_437a_a5ed_9daecd1a882a.slice/crio-41458720d31addb1351e0b95ef3c56b790b76e48b120423359b73bdc5e2756ab WatchSource:0}: Error finding container 41458720d31addb1351e0b95ef3c56b790b76e48b120423359b73bdc5e2756ab: Status 404 returned error can't find the container with id 41458720d31addb1351e0b95ef3c56b790b76e48b120423359b73bdc5e2756ab
Nov 02 18:23:22.390419 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:22.390378    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:23:22.631056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:22.631023    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j" event={"ID":"4d7310ab-a339-437a-a5ed-9daecd1a882a","Type":"ContainerStarted","Data":"41458720d31addb1351e0b95ef3c56b790b76e48b120423359b73bdc5e2756ab"}
Nov 02 18:23:23.640004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:23.639970    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j" event={"ID":"4d7310ab-a339-437a-a5ed-9daecd1a882a","Type":"ContainerStarted","Data":"314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a"}
Nov 02 18:23:23.640426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:23.640036    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:23.758259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:23.758206    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j" podStartSLOduration=1.971032082 podStartE2EDuration="2.758191761s" totalImagesPullingTime="787.159679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:23.754804046 +0000 UTC m=+3634.011834677" watchObservedRunningTime="2026-11-02 18:23:23.758191761 +0000 UTC m=+3634.015222379"
Nov 02 18:23:29.057715 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:23:29.057678    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:23:29.652203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:29.652101    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:23:59.054303 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:23:59.054268    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:23:59.539292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:59.539259    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"]
Nov 02 18:23:59.539533 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:23:59.539456    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" containerID="cri-o://235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797" gracePeriod=30
Nov 02 18:24:00.051348 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:00.051310    4956 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:24:00.051546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:00.051364    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:24:04.812309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.812234    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:24:04.831446 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831411    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-data\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831596 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831465    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-envoy\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831596 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831535    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-kube-api-access-n9t6f\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-kube-api-access-n9t6f\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831596 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831560    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-podinfo\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831602    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-socket\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831640    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-token\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-token\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831710    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-data" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:04.831738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831729    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-credential-socket\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831757    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istiod-ca-cert\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831833    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-certs\") pod \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\" (UID: \"c3572bea-8ee7-4bf6-8623-65372e8ad09f\") "
Nov 02 18:24:04.831949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.831934    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-socket" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:04.832153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.832048    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-credential-socket" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:04.832301 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.832281    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istiod-ca-cert" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:04.832442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.832421    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.832517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.832450    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.832517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.832464    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.832517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.832478    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.832517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.832461    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-certs" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:04.833956 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.833935    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-kube-api-access-n9t6f" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "kube-api-access-n9t6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:04.834068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.833950    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-token" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:04.834347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.834329    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-envoy" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:04.834411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.834352    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-podinfo" pod "c3572bea-8ee7-4bf6-8623-65372e8ad09f" (UID: "c3572bea-8ee7-4bf6-8623-65372e8ad09f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:24:04.904282 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.904249    4956 generic.go:356] "Generic (PLEG): container finished" podID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerID="235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797" exitCode=0
Nov 02 18:24:04.904464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.904299    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" event={"ID":"c3572bea-8ee7-4bf6-8623-65372e8ad09f","Type":"ContainerDied","Data":"235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797"}
Nov 02 18:24:04.904464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.904338    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs" event={"ID":"c3572bea-8ee7-4bf6-8623-65372e8ad09f","Type":"ContainerDied","Data":"a031fd1871423144962bd9f43f0250d7cf9b1412ef557365514ffdbf7b99cbb9"}
Nov 02 18:24:04.904464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.904370    4956 scope.go:122] "RemoveContainer" containerID="235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797"
Nov 02 18:24:04.904464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.904301    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"
Nov 02 18:24:04.915391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.915373    4956 scope.go:122] "RemoveContainer" containerID="235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797"
Nov 02 18:24:04.915694 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:24:04.915673    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797\": container with ID starting with 235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797 not found: ID does not exist" containerID="235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797"
Nov 02 18:24:04.915737 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.915704    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797"} err="failed to get container status \"235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797\": rpc error: code = NotFound desc = could not find container \"235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797\": container with ID starting with 235c6b21e5eb8ac1adee8b06d5d551de06b1985d8cee766d27797c14f3d37797 not found: ID does not exist"
Nov 02 18:24:04.933769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.933740    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.933769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.933767    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.933947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.933780    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9t6f\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-kube-api-access-n9t6f\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.933947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.933792    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:04.933947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:04.933802    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3572bea-8ee7-4bf6-8623-65372e8ad09f-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:05.093351 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:05.093258    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"]
Nov 02 18:24:05.096517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:05.096486    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-b74vs"]
Nov 02 18:24:05.866215 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:05.866180    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" path="/var/lib/kubelet/pods/c3572bea-8ee7-4bf6-8623-65372e8ad09f/volumes"
Nov 02 18:24:11.498485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:11.498444    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"]
Nov 02 18:24:16.240200 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.240162    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"]
Nov 02 18:24:16.349667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.349626    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"]
Nov 02 18:24:16.499579 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.499488    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd" podUID="1b4c0f49-197e-44c8-ac7f-016c175044ff" containerName="payload-pre-processing" containerID="cri-o://aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130" gracePeriod=25
Nov 02 18:24:16.776663 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.776640    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:24:16.897261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.897227    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4c0f49-197e-44c8-ac7f-016c175044ff-kube-api-access-m2zbm\" (UniqueName: \"kubernetes.io/projected/1b4c0f49-197e-44c8-ac7f-016c175044ff-kube-api-access-m2zbm\") pod \"1b4c0f49-197e-44c8-ac7f-016c175044ff\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") "
Nov 02 18:24:16.897451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.897433    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4c0f49-197e-44c8-ac7f-016c175044ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4c0f49-197e-44c8-ac7f-016c175044ff-plugins-config-volume\") pod \"1b4c0f49-197e-44c8-ac7f-016c175044ff\" (UID: \"1b4c0f49-197e-44c8-ac7f-016c175044ff\") "
Nov 02 18:24:16.897796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.897769    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4c0f49-197e-44c8-ac7f-016c175044ff-plugins-config-volume" pod "1b4c0f49-197e-44c8-ac7f-016c175044ff" (UID: "1b4c0f49-197e-44c8-ac7f-016c175044ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:16.897935 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.897893    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4c0f49-197e-44c8-ac7f-016c175044ff-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:16.901534 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.901508    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4c0f49-197e-44c8-ac7f-016c175044ff-kube-api-access-m2zbm" pod "1b4c0f49-197e-44c8-ac7f-016c175044ff" (UID: "1b4c0f49-197e-44c8-ac7f-016c175044ff"). InnerVolumeSpecName "kube-api-access-m2zbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:16.987227 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.987198    4956 generic.go:356] "Generic (PLEG): container finished" podID="1b4c0f49-197e-44c8-ac7f-016c175044ff" containerID="aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130" exitCode=0
Nov 02 18:24:16.987404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.987245    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"
Nov 02 18:24:16.987404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.987262    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd" event={"ID":"1b4c0f49-197e-44c8-ac7f-016c175044ff","Type":"ContainerDied","Data":"aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130"}
Nov 02 18:24:16.987404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.987304    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd" event={"ID":"1b4c0f49-197e-44c8-ac7f-016c175044ff","Type":"ContainerDied","Data":"1ed0f0c2246f0bbf7e81787fa3f15d7bf9be6540f775c242198c6396cd9ef658"}
Nov 02 18:24:16.987404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.987325    4956 scope.go:122] "RemoveContainer" containerID="aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130"
Nov 02 18:24:16.997978 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.997961    4956 scope.go:122] "RemoveContainer" containerID="aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130"
Nov 02 18:24:16.998316 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:24:16.998291    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130\": container with ID starting with aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130 not found: ID does not exist" containerID="aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130"
Nov 02 18:24:16.998416 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.998324    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130"} err="failed to get container status \"aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130\": rpc error: code = NotFound desc = could not find container \"aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130\": container with ID starting with aea5fc4ee83229ded6e2931cbe153aaa10236a0dfe015037bbfb72a8dacc7130 not found: ID does not exist"
Nov 02 18:24:16.998629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:16.998608    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2zbm\" (UniqueName: \"kubernetes.io/projected/1b4c0f49-197e-44c8-ac7f-016c175044ff-kube-api-access-m2zbm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:17.153566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:17.153476    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"]
Nov 02 18:24:17.157384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:17.157359    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-508b8f-7b7685ff6c-2gcjd"]
Nov 02 18:24:17.866369 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:17.866339    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4c0f49-197e-44c8-ac7f-016c175044ff" path="/var/lib/kubelet/pods/1b4c0f49-197e-44c8-ac7f-016c175044ff/volumes"
Nov 02 18:24:21.241280 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.241238    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg" podUID="dcccd037-a971-4c7f-974f-9f2b95da6fe2" containerName="payload-processing" containerID="cri-o://dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663" gracePeriod=25
Nov 02 18:24:21.350879 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.350835    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2" podUID="faa577e5-a13b-4d2b-b610-582bf0ed01d5" containerName="payload-pre-processing" containerID="cri-o://23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365" gracePeriod=25
Nov 02 18:24:21.533433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.533401    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:24:21.646654 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.646629    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:24:21.657791 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.657761    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcccd037-a971-4c7f-974f-9f2b95da6fe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcccd037-a971-4c7f-974f-9f2b95da6fe2-plugins-config-volume\") pod \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") "
Nov 02 18:24:21.657975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.657921    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcccd037-a971-4c7f-974f-9f2b95da6fe2-kube-api-access-8trzs\" (UniqueName: \"kubernetes.io/projected/dcccd037-a971-4c7f-974f-9f2b95da6fe2-kube-api-access-8trzs\") pod \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\" (UID: \"dcccd037-a971-4c7f-974f-9f2b95da6fe2\") "
Nov 02 18:24:21.658959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.658933    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcccd037-a971-4c7f-974f-9f2b95da6fe2-plugins-config-volume" pod "dcccd037-a971-4c7f-974f-9f2b95da6fe2" (UID: "dcccd037-a971-4c7f-974f-9f2b95da6fe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:21.661437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.661409    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcccd037-a971-4c7f-974f-9f2b95da6fe2-kube-api-access-8trzs" pod "dcccd037-a971-4c7f-974f-9f2b95da6fe2" (UID: "dcccd037-a971-4c7f-974f-9f2b95da6fe2"). InnerVolumeSpecName "kube-api-access-8trzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:21.760548 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.759627    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa577e5-a13b-4d2b-b610-582bf0ed01d5-kube-api-access-6d997\" (UniqueName: \"kubernetes.io/projected/faa577e5-a13b-4d2b-b610-582bf0ed01d5-kube-api-access-6d997\") pod \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") "
Nov 02 18:24:21.760548 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.759687    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa577e5-a13b-4d2b-b610-582bf0ed01d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa577e5-a13b-4d2b-b610-582bf0ed01d5-plugins-config-volume\") pod \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\" (UID: \"faa577e5-a13b-4d2b-b610-582bf0ed01d5\") "
Nov 02 18:24:21.760775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.760574    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa577e5-a13b-4d2b-b610-582bf0ed01d5-plugins-config-volume" pod "faa577e5-a13b-4d2b-b610-582bf0ed01d5" (UID: "faa577e5-a13b-4d2b-b610-582bf0ed01d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:21.761404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.761372    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcccd037-a971-4c7f-974f-9f2b95da6fe2-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:21.761404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.761405    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa577e5-a13b-4d2b-b610-582bf0ed01d5-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:21.761617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.761419    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trzs\" (UniqueName: \"kubernetes.io/projected/dcccd037-a971-4c7f-974f-9f2b95da6fe2-kube-api-access-8trzs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:21.764051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.764019    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa577e5-a13b-4d2b-b610-582bf0ed01d5-kube-api-access-6d997" pod "faa577e5-a13b-4d2b-b610-582bf0ed01d5" (UID: "faa577e5-a13b-4d2b-b610-582bf0ed01d5"). InnerVolumeSpecName "kube-api-access-6d997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:21.862476 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:21.862440    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d997\" (UniqueName: \"kubernetes.io/projected/faa577e5-a13b-4d2b-b610-582bf0ed01d5-kube-api-access-6d997\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.025415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.025322    4956 generic.go:356] "Generic (PLEG): container finished" podID="faa577e5-a13b-4d2b-b610-582bf0ed01d5" containerID="23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365" exitCode=0
Nov 02 18:24:22.025415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.025387    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"
Nov 02 18:24:22.025635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.025410    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2" event={"ID":"faa577e5-a13b-4d2b-b610-582bf0ed01d5","Type":"ContainerDied","Data":"23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365"}
Nov 02 18:24:22.025635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.025476    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2" event={"ID":"faa577e5-a13b-4d2b-b610-582bf0ed01d5","Type":"ContainerDied","Data":"53e0743da30301af74b61301f45ca0ed54b98f7e78453a6e3b355fa928645b68"}
Nov 02 18:24:22.025635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.025510    4956 scope.go:122] "RemoveContainer" containerID="23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365"
Nov 02 18:24:22.026985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.026959    4956 generic.go:356] "Generic (PLEG): container finished" podID="dcccd037-a971-4c7f-974f-9f2b95da6fe2" containerID="dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663" exitCode=0
Nov 02 18:24:22.027100 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.027059    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg" event={"ID":"dcccd037-a971-4c7f-974f-9f2b95da6fe2","Type":"ContainerDied","Data":"dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663"}
Nov 02 18:24:22.027100 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.027095    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg" event={"ID":"dcccd037-a971-4c7f-974f-9f2b95da6fe2","Type":"ContainerDied","Data":"25bfe28738d6be2f68452a96082e66fdb224c6a3378177e791a0fda7489434fa"}
Nov 02 18:24:22.028331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.028310    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"
Nov 02 18:24:22.036320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.036295    4956 scope.go:122] "RemoveContainer" containerID="23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365"
Nov 02 18:24:22.036611 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:24:22.036594    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365\": container with ID starting with 23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365 not found: ID does not exist" containerID="23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365"
Nov 02 18:24:22.036662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.036619    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365"} err="failed to get container status \"23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365\": rpc error: code = NotFound desc = could not find container \"23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365\": container with ID starting with 23cf5c57c97b6e443ef6e8d6449c0c5dcdbd00c562115e5ac8d1d9c666a97365 not found: ID does not exist"
Nov 02 18:24:22.036662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.036636    4956 scope.go:122] "RemoveContainer" containerID="dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663"
Nov 02 18:24:22.045925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.045906    4956 scope.go:122] "RemoveContainer" containerID="dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663"
Nov 02 18:24:22.046324 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:24:22.046306    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663\": container with ID starting with dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663 not found: ID does not exist" containerID="dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663"
Nov 02 18:24:22.046393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.046335    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663"} err="failed to get container status \"dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663\": rpc error: code = NotFound desc = could not find container \"dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663\": container with ID starting with dc1663d54f81cc00e665ce5ade149c017bcbe2bd8c53ffdc90633b4186253663 not found: ID does not exist"
Nov 02 18:24:22.193017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.192986    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"]
Nov 02 18:24:22.242488 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.242457    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2"]
Nov 02 18:24:22.304684 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.304585    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"]
Nov 02 18:24:22.308381 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.308339    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a8da6e-5fc495dd69-8kbhg"]
Nov 02 18:24:22.557767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:22.557667    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8da6e-7f44b87bf5-vffv2" podUID="faa577e5-a13b-4d2b-b610-582bf0ed01d5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.146:50052\" within 1s: context deadline exceeded"
Nov 02 18:24:22.557767 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:24:22.557742    4956 logging.go:55] [core] [Channel #10057 SubChannel #10058]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.146:50052", ServerName: "10.132.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.146:50052: operation was canceled"
Nov 02 18:24:23.866690 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:23.866652    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcccd037-a971-4c7f-974f-9f2b95da6fe2" path="/var/lib/kubelet/pods/dcccd037-a971-4c7f-974f-9f2b95da6fe2/volumes"
Nov 02 18:24:23.867297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:23.867277    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa577e5-a13b-4d2b-b610-582bf0ed01d5" path="/var/lib/kubelet/pods/faa577e5-a13b-4d2b-b610-582bf0ed01d5/volumes"
Nov 02 18:24:29.077067 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:24:29.077025    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:24:33.377734 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.377706    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"]
Nov 02 18:24:33.378550 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378528    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1b4c0f49-197e-44c8-ac7f-016c175044ff" containerName="payload-pre-processing"
Nov 02 18:24:33.378550 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378548    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4c0f49-197e-44c8-ac7f-016c175044ff" containerName="payload-pre-processing"
Nov 02 18:24:33.378697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378564    4956 cpu_manager.go:414] "containerMap: removing container" podUID="faa577e5-a13b-4d2b-b610-582bf0ed01d5" containerName="payload-pre-processing"
Nov 02 18:24:33.378697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378570    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa577e5-a13b-4d2b-b610-582bf0ed01d5" containerName="payload-pre-processing"
Nov 02 18:24:33.378697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378590    4956 cpu_manager.go:414] "containerMap: removing container" podUID="dcccd037-a971-4c7f-974f-9f2b95da6fe2" containerName="payload-processing"
Nov 02 18:24:33.378697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378595    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcccd037-a971-4c7f-974f-9f2b95da6fe2" containerName="payload-processing"
Nov 02 18:24:33.378697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378619    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy"
Nov 02 18:24:33.378697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378624    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy"
Nov 02 18:24:33.378881 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378715    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3572bea-8ee7-4bf6-8623-65372e8ad09f" containerName="istio-proxy"
Nov 02 18:24:33.378881 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378726    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4c0f49-197e-44c8-ac7f-016c175044ff" containerName="payload-pre-processing"
Nov 02 18:24:33.378881 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378732    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa577e5-a13b-4d2b-b610-582bf0ed01d5" containerName="payload-pre-processing"
Nov 02 18:24:33.378881 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.378740    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcccd037-a971-4c7f-974f-9f2b95da6fe2" containerName="payload-processing"
Nov 02 18:24:33.381091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.381075    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.531917    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532036    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b1abdf-ba68-4f11-a240-93914229abcd-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532100    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532187    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b1abdf-ba68-4f11-a240-93914229abcd-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532280    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532338    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532371    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532403    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.532595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.532553    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqs2g\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-kube-api-access-dqs2g\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.589797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.589761    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"]
Nov 02 18:24:33.633750 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.633654    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.633750 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.633718    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b1abdf-ba68-4f11-a240-93914229abcd-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.633994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.633771    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.633994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.633786    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b1abdf-ba68-4f11-a240-93914229abcd-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.633994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.633882    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.633994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.633949    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.633996    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.634045    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.634163    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqs2g\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-kube-api-access-dqs2g\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634374 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.634313    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.634399    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634501 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.634471    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.634609    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.634908 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.634884    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b1abdf-ba68-4f11-a240-93914229abcd-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.636197 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.636172    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.636557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.636535    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b1abdf-ba68-4f11-a240-93914229abcd-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.691708 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.691674    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.691874 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.691833    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqs2g\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-kube-api-access-dqs2g\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:33.753951 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:33.753903    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:34.118605 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.118568    4956 generic.go:356] "Generic (PLEG): container finished" podID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerID="1fab816052b0b3334280ac04544fb3de250d859b353d5595f39d2e4983bd0354" exitCode=0
Nov 02 18:24:34.118814 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.118645    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" event={"ID":"3f7d4c4c-2363-4cd5-903b-10832fe15b8a","Type":"ContainerDied","Data":"1fab816052b0b3334280ac04544fb3de250d859b353d5595f39d2e4983bd0354"}
Nov 02 18:24:34.119071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.119054    4956 scope.go:122] "RemoveContainer" containerID="1fab816052b0b3334280ac04544fb3de250d859b353d5595f39d2e4983bd0354"
Nov 02 18:24:34.120617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.120593    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:24:34.120701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.120659    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:24:34.120701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.120693    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:24:34.186365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.186327    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"]
Nov 02 18:24:34.195433 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:24:34.195394    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b1abdf_ba68_4f11_a240_93914229abcd.slice/crio-06973fd9f06deece9486ad477a48dec924266d34d5ac9dddecc7d7d7fae4f4d9 WatchSource:0}: Error finding container 06973fd9f06deece9486ad477a48dec924266d34d5ac9dddecc7d7d7fae4f4d9: Status 404 returned error can't find the container with id 06973fd9f06deece9486ad477a48dec924266d34d5ac9dddecc7d7d7fae4f4d9
Nov 02 18:24:34.199424 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.199379    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:24:34.199617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.199473    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:24:34.199617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.199514    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:24:34.690538 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:34.690504    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:24:35.148146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.143148    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" event={"ID":"75b1abdf-ba68-4f11-a240-93914229abcd","Type":"ContainerStarted","Data":"8e0e7fa529d52611f26a3fff4f09f0fc0f0492d88bb0ba4a1908e64e16af81be"}
Nov 02 18:24:35.148146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.143225    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" event={"ID":"75b1abdf-ba68-4f11-a240-93914229abcd","Type":"ContainerStarted","Data":"06973fd9f06deece9486ad477a48dec924266d34d5ac9dddecc7d7d7fae4f4d9"}
Nov 02 18:24:35.148146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.146807    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" event={"ID":"3f7d4c4c-2363-4cd5-903b-10832fe15b8a","Type":"ContainerStarted","Data":"529dac07f55c5cf14875d9c8d72ebeaa1fc149a36a2bf374665d5b6dbc0fa47a"}
Nov 02 18:24:35.286648 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.285686    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" podStartSLOduration=2.285669366 podStartE2EDuration="2.285669366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:24:35.271679149 +0000 UTC m=+3705.528709770" watchObservedRunningTime="2026-11-02 18:24:35.285669366 +0000 UTC m=+3705.542699986"
Nov 02 18:24:35.759145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.755534    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:35.769145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.767266    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 18:24:35.769145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.767341    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 18:24:35.953147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.951539    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:24:35.974148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.964293    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 18:24:35.974148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:35.964376    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 18:24:36.758622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:36.758580    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 18:24:36.759279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:36.759245    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 18:24:36.930278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:36.930232    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 18:24:36.931288 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:36.931254    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 18:24:37.755148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:37.754608    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 18:24:37.755148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:37.754681    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 18:24:37.938161 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:37.937570    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 18:24:37.938161 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:37.937638    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 18:24:38.761149 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:38.757579    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 18:24:38.761149 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:38.757655    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 18:24:38.940156 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:38.939625    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 18:24:38.940156 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:38.939703    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 18:24:39.771150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:39.761603    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 18:24:39.771150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:39.761672    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 18:24:39.948144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:39.947363    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 18:24:39.948144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:39.947426    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 18:24:40.759808 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:40.759778    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:40.760279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:40.760254    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:40.761152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:40.761108    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:24:40.935872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:40.935840    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:24:40.936160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:40.936138    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:24:40.936937 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:24:40.936919    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:24:59.053951 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:24:59.053913    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:25:03.630684 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.630604    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"]
Nov 02 18:25:03.634880 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.634860    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673263    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673323    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673376    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673458    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673600 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673575    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af46991d-98e5-4c24-a847-17a0a8b3b07a-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673638 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673617    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673674 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673648    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673753 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673741    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szsch\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-kube-api-access-szsch\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.673798 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.673784    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775373    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af46991d-98e5-4c24-a847-17a0a8b3b07a-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775439    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775479    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775519    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szsch\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-kube-api-access-szsch\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775567    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775604    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775640    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775667    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.775977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.775706    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.776842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.776814    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af46991d-98e5-4c24-a847-17a0a8b3b07a-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.777099 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.777081    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.777677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.777654    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.778071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.778035    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.778268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.778237    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.779541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.779510    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.780482 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.780457    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.832538 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.832505    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.836642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.836615    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szsch\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-kube-api-access-szsch\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:03.837817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.837791    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"]
Nov 02 18:25:03.949706 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:03.949616    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:04.387744 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:04.387713    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"]
Nov 02 18:25:04.388496 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:25:04.388472    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf46991d_98e5_4c24_a847_17a0a8b3b07a.slice/crio-3a9318f136897ae1164b825140f36bb798d61baea9d32a2c1ff38b05245a3fcc WatchSource:0}: Error finding container 3a9318f136897ae1164b825140f36bb798d61baea9d32a2c1ff38b05245a3fcc: Status 404 returned error can't find the container with id 3a9318f136897ae1164b825140f36bb798d61baea9d32a2c1ff38b05245a3fcc
Nov 02 18:25:04.391112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:04.391082    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:04.391208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:04.391172    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:04.391270 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:04.391223    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:05.387458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:05.387420    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" event={"ID":"af46991d-98e5-4c24-a847-17a0a8b3b07a","Type":"ContainerStarted","Data":"c22c4b8cb9859d1c2a716063d5f1e4c799cb97b695d4570ff50fad39d6d2e855"}
Nov 02 18:25:05.387886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:05.387483    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" event={"ID":"af46991d-98e5-4c24-a847-17a0a8b3b07a","Type":"ContainerStarted","Data":"3a9318f136897ae1164b825140f36bb798d61baea9d32a2c1ff38b05245a3fcc"}
Nov 02 18:25:05.614641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:05.614580    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" podStartSLOduration=2.6145649669999997 podStartE2EDuration="2.614564967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:05.562187296 +0000 UTC m=+3735.819217917" watchObservedRunningTime="2026-11-02 18:25:05.614564967 +0000 UTC m=+3735.871595586"
Nov 02 18:25:05.950294 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:05.950255    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:05.952382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:05.952353    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:25:05.952500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:05.952415    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 18:25:06.950609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:06.950564    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:25:06.951201 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:06.950627    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 18:25:07.950992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:07.950947    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:25:07.951547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:07.951010    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 18:25:08.950267 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:08.950224    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:25:08.950495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:08.950283    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 18:25:09.950570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:09.950526    4956 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:25:09.951159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:09.950588    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 18:25:10.955736 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:10.955699    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:10.956240 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:10.956222    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:10.957009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:10.956987    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:25:29.052275 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:25:29.052237    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:25:53.313810 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:53.313770    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"]
Nov 02 18:25:53.314241 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:53.313981    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy" containerID="cri-o://5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628" gracePeriod=30
Nov 02 18:25:58.596077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.596004    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:25:58.689336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689300    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-data\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689342    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-podinfo\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689367    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-envoy\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689446    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bea785-b1bf-45cd-bd9d-63678fb98a07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bea785-b1bf-45cd-bd9d-63678fb98a07-istiod-ca-cert\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689492    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-token\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-token\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689544    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-credential-socket\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689585    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-certs\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689617    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-kube-api-access-4xnhg\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-kube-api-access-4xnhg\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.689795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689626    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-data" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:25:58.689795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689770    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-socket\") pod \"10bea785-b1bf-45cd-bd9d-63678fb98a07\" (UID: \"10bea785-b1bf-45cd-bd9d-63678fb98a07\") "
Nov 02 18:25:58.690009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689894    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-certs" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:25:58.690009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.689940    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bea785-b1bf-45cd-bd9d-63678fb98a07-istiod-ca-cert" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:58.690201 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.690178    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-socket" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:25:58.690288 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.690209    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-credential-socket" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:25:58.690497 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.690479    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.690589 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.690503    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bea785-b1bf-45cd-bd9d-63678fb98a07-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.690589 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.690522    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.690589 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.690536    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.690589 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.690550    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.691941 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.691915    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-podinfo" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:25:58.692006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.691953    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-token" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:58.692486 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.692469    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-kube-api-access-4xnhg" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "kube-api-access-4xnhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:58.692553 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.692500    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-envoy" pod "10bea785-b1bf-45cd-bd9d-63678fb98a07" (UID: "10bea785-b1bf-45cd-bd9d-63678fb98a07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:25:58.743636 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.743600    4956 generic.go:356] "Generic (PLEG): container finished" podID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerID="5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628" exitCode=0
Nov 02 18:25:58.743813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.743649    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"
Nov 02 18:25:58.743813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.743687    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" event={"ID":"10bea785-b1bf-45cd-bd9d-63678fb98a07","Type":"ContainerDied","Data":"5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628"}
Nov 02 18:25:58.743813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.743747    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8" event={"ID":"10bea785-b1bf-45cd-bd9d-63678fb98a07","Type":"ContainerDied","Data":"ccc509a3e7b125a53380772f5c3835233fbb1d4383a0667866382d135bc68ab7"}
Nov 02 18:25:58.743813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.743769    4956 scope.go:122] "RemoveContainer" containerID="5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628"
Nov 02 18:25:58.755907 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.755875    4956 scope.go:122] "RemoveContainer" containerID="fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b"
Nov 02 18:25:58.766696 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.766673    4956 scope.go:122] "RemoveContainer" containerID="5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628"
Nov 02 18:25:58.767090 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:25:58.767069    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628\": container with ID starting with 5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628 not found: ID does not exist" containerID="5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628"
Nov 02 18:25:58.767205 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.767103    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628"} err="failed to get container status \"5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628\": rpc error: code = NotFound desc = could not find container \"5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628\": container with ID starting with 5ed5576e7cad26b88ecd30d32d561a8fe18c6114629ef03f0371c2dd231c0628 not found: ID does not exist"
Nov 02 18:25:58.767205 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.767152    4956 scope.go:122] "RemoveContainer" containerID="fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b"
Nov 02 18:25:58.767548 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:25:58.767529    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b\": container with ID starting with fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b not found: ID does not exist" containerID="fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b"
Nov 02 18:25:58.767638 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.767554    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b"} err="failed to get container status \"fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b\": rpc error: code = NotFound desc = could not find container \"fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b\": container with ID starting with fc428b273d9f05d2cd3f89636fc8d2e3d60d143ac5f4c90f4ee10376089f227b not found: ID does not exist"
Nov 02 18:25:58.791735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.791699    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.791735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.791727    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.791735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.791739    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.792047 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.791748    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xnhg\" (UniqueName: \"kubernetes.io/projected/10bea785-b1bf-45cd-bd9d-63678fb98a07-kube-api-access-4xnhg\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:58.925874 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.925782    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"]
Nov 02 18:25:58.930702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:58.930671    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-508b8f-openshift-default-768f765bcc-hkjz8"]
Nov 02 18:25:59.055378 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:25:59.055332    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:25:59.867228 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:25:59.867197    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" path="/var/lib/kubelet/pods/10bea785-b1bf-45cd-bd9d-63678fb98a07/volumes"
Nov 02 18:26:00.543512 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.543483    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"]
Nov 02 18:26:00.544108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.544095    4956 cpu_manager.go:414] "containerMap: removing container" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy"
Nov 02 18:26:00.544185 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.544110    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy"
Nov 02 18:26:00.544276 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.544266    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy"
Nov 02 18:26:00.544312 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.544280    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy"
Nov 02 18:26:00.549208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.549185    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.651163 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.651099    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4bef29666866\""
Nov 02 18:26:00.651372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.651099    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 18:26:00.651372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.651111    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ad0e4981cd4f\""
Nov 02 18:26:00.651372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.651111    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 18:26:00.651372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.651208    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 18:26:00.651372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.651208    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 18:26:00.651372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.651262    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n74nw\""
Nov 02 18:26:00.654335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.654314    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"]
Nov 02 18:26:00.715001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.714958    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-plugins-config-volume\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.715230 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.715199    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspt4\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-kube-api-access-mspt4\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.715309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.715293    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.715348 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.715325    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-routing-overlay-volume\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.816262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.816165    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.816262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.816209    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-routing-overlay-volume\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.816502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.816281    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-plugins-config-volume\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.816502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.816428    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspt4\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-kube-api-access-mspt4\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.817023 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.816990    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-routing-overlay-volume\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.817138 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.817098    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-plugins-config-volume\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.819414 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.819391    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:00.874382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:00.874341    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspt4\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-kube-api-access-mspt4\") pod \"payload-processing-external-model-8ddf7bb45-qzn7w\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") " pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:01.163943 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:01.163859    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:01.549003 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:26:01.548969    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9a4a34_abee_461e_a01a_3f1f89ba22eb.slice/crio-8b203eef1b0c7b0b675bfb75bfc53df0ea9dddc5c7345175c1ac6af759a54dab WatchSource:0}: Error finding container 8b203eef1b0c7b0b675bfb75bfc53df0ea9dddc5c7345175c1ac6af759a54dab: Status 404 returned error can't find the container with id 8b203eef1b0c7b0b675bfb75bfc53df0ea9dddc5c7345175c1ac6af759a54dab
Nov 02 18:26:01.549438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:01.549415    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"]
Nov 02 18:26:01.765888 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:01.765847    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" event={"ID":"cd9a4a34-abee-461e-a01a-3f1f89ba22eb","Type":"ContainerStarted","Data":"e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464"}
Nov 02 18:26:01.766077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:01.765910    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" event={"ID":"cd9a4a34-abee-461e-a01a-3f1f89ba22eb","Type":"ContainerStarted","Data":"8b203eef1b0c7b0b675bfb75bfc53df0ea9dddc5c7345175c1ac6af759a54dab"}
Nov 02 18:26:01.766077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:01.765940    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:01.885642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:01.885535    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" podStartSLOduration=1.885518261 podStartE2EDuration="1.885518261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:01.882115074 +0000 UTC m=+3792.139145704" watchObservedRunningTime="2026-11-02 18:26:01.885518261 +0000 UTC m=+3792.142548899"
Nov 02 18:26:10.392390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.392353    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"]
Nov 02 18:26:10.393149 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.393095    4956 cpu_manager.go:414] "containerMap: removing container" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy"
Nov 02 18:26:10.393149 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.393132    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bea785-b1bf-45cd-bd9d-63678fb98a07" containerName="istio-proxy"
Nov 02 18:26:10.396999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.396980    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438257    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438299    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedc2ef6-768a-4464-8292-4363506a59f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438549 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438469    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438549 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438508    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438549 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438540    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95q7q\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-kube-api-access-95q7q\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438674 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438556    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438674 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438628    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedc2ef6-768a-4464-8292-4363506a59f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438741 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438674    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.438778 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.438759    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.449196 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.449173    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\""
Nov 02 18:26:10.503316 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.503272    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"]
Nov 02 18:26:10.545311 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.545262    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.545611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.545589    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.545778 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.545760    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95q7q\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-kube-api-access-95q7q\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.545897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.545883    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.546005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.545992    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedc2ef6-768a-4464-8292-4363506a59f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.546189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.546171    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.546356 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.546342    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.546540 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.546510    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.546673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.546557    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedc2ef6-768a-4464-8292-4363506a59f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.548259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.548229    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.548519 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.548497    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.548827 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.548802    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.549054 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.549032    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.549570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.549546    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedc2ef6-768a-4464-8292-4363506a59f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.550186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.550155    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedc2ef6-768a-4464-8292-4363506a59f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.555661 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.555628    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.605217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.605188    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.605368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.605248    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95q7q\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-kube-api-access-95q7q\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-46pcb\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:10.712156 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:10.712098    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:11.154647 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:11.154615    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"]
Nov 02 18:26:11.155388 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:26:11.155358    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedc2ef6_768a_4464_8292_4363506a59f2.slice/crio-46bba3aea2854f19eb2dcc8c4878d3ea15f5649096d3a31dcf3a98ab234c500a WatchSource:0}: Error finding container 46bba3aea2854f19eb2dcc8c4878d3ea15f5649096d3a31dcf3a98ab234c500a: Status 404 returned error can't find the container with id 46bba3aea2854f19eb2dcc8c4878d3ea15f5649096d3a31dcf3a98ab234c500a
Nov 02 18:26:11.158050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:11.158020    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:11.158110 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:11.158086    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:11.158168 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:11.158155    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:11.844533 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:11.844487    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" event={"ID":"aedc2ef6-768a-4464-8292-4363506a59f2","Type":"ContainerStarted","Data":"e373e7c36aca97ed73c8cbdee2f32c200121d96c6b41a51f06966ad6d1b05c4d"}
Nov 02 18:26:11.844977 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:11.844566    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" event={"ID":"aedc2ef6-768a-4464-8292-4363506a59f2","Type":"ContainerStarted","Data":"46bba3aea2854f19eb2dcc8c4878d3ea15f5649096d3a31dcf3a98ab234c500a"}
Nov 02 18:26:12.064880 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:12.064818    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podStartSLOduration=2.06480307 podStartE2EDuration="2.06480307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:12.013012549 +0000 UTC m=+3802.270043169" watchObservedRunningTime="2026-11-02 18:26:12.06480307 +0000 UTC m=+3802.321833690"
Nov 02 18:26:12.713035 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:12.712997    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:12.714584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:12.714551    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:26:12.714750 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:12.714604    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 18:26:12.777789 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:12.777759    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:26:13.712527 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:13.712492    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:26:13.712971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:13.712548    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 18:26:14.712542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:14.712502    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:26:14.712967 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:14.712574    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 18:26:15.714081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:15.714048    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:26:15.714751 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:15.714142    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 18:26:16.713525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:16.713487    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:26:16.713739 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:16.713568    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 18:26:17.713385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:17.713351    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:26:17.713947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:17.713412    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 18:26:18.712754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:18.712715    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:26:18.712939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:18.712786    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 18:26:19.846616 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:19.846582    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:19.848045 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:19.848017    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:19.848848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:19.848826    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:26:29.053086 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:29.053045    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:26:35.336894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.336808    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"]
Nov 02 18:26:35.341613 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.341590    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.446817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.446785    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"]
Nov 02 18:26:35.490953 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.490914    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmv5r\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-kube-api-access-tmv5r\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.491182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.490975    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.491182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.491090    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.491182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.491168    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.592511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.592417    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmv5r\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-kube-api-access-tmv5r\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.592511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.592469    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.592767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.592520    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.592767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.592542    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.593249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.593223    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.593381 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.593270    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.595354 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.595329    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.649900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.649863    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmv5r\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-kube-api-access-tmv5r\") pod \"payload-processing-external-model-596fb6c5c4-rfs6p\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:35.657745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:35.657715    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:36.039026 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:26:36.038997    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f85f8d_1361_48e1_986c_9495172f4da4.slice/crio-6b848057e7cac9d619fdbc71fa1c137448a04e7000af0bffe690a744884e6f2a WatchSource:0}: Error finding container 6b848057e7cac9d619fdbc71fa1c137448a04e7000af0bffe690a744884e6f2a: Status 404 returned error can't find the container with id 6b848057e7cac9d619fdbc71fa1c137448a04e7000af0bffe690a744884e6f2a
Nov 02 18:26:36.039172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:36.039020    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"]
Nov 02 18:26:37.023186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:37.023148    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p" event={"ID":"c1f85f8d-1361-48e1-986c-9495172f4da4","Type":"ContainerStarted","Data":"ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9"}
Nov 02 18:26:37.023643 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:37.023209    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p" event={"ID":"c1f85f8d-1361-48e1-986c-9495172f4da4","Type":"ContainerStarted","Data":"6b848057e7cac9d619fdbc71fa1c137448a04e7000af0bffe690a744884e6f2a"}
Nov 02 18:26:37.023643 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:37.023264    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:37.139442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:37.139390    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p" podStartSLOduration=2.139375065 podStartE2EDuration="2.139375065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:37.136736471 +0000 UTC m=+3827.393767090" watchObservedRunningTime="2026-11-02 18:26:37.139375065 +0000 UTC m=+3827.396405685"
Nov 02 18:26:40.557651 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:40.557619    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"]
Nov 02 18:26:45.557930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.557881    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p" podUID="c1f85f8d-1361-48e1-986c-9495172f4da4" containerName="payload-processing" containerID="cri-o://ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9" gracePeriod=25
Nov 02 18:26:45.831647 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.831620    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:45.958584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.958544    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-provider-credentials\") pod \"c1f85f8d-1361-48e1-986c-9495172f4da4\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") "
Nov 02 18:26:45.958796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.958597    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-routing-overlay-volume\") pod \"c1f85f8d-1361-48e1-986c-9495172f4da4\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") "
Nov 02 18:26:45.958881 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.958809    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-plugins-config-volume\") pod \"c1f85f8d-1361-48e1-986c-9495172f4da4\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") "
Nov 02 18:26:45.959385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.959066    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-kube-api-access-tmv5r\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-kube-api-access-tmv5r\") pod \"c1f85f8d-1361-48e1-986c-9495172f4da4\" (UID: \"c1f85f8d-1361-48e1-986c-9495172f4da4\") "
Nov 02 18:26:45.959385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.959250    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-routing-overlay-volume" pod "c1f85f8d-1361-48e1-986c-9495172f4da4" (UID: "c1f85f8d-1361-48e1-986c-9495172f4da4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:26:45.959562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.959389    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-plugins-config-volume" pod "c1f85f8d-1361-48e1-986c-9495172f4da4" (UID: "c1f85f8d-1361-48e1-986c-9495172f4da4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:26:45.959970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.959945    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:45.960072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.959974    4956 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85f8d-1361-48e1-986c-9495172f4da4-routing-overlay-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:45.961425 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.961399    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-kube-api-access-tmv5r" pod "c1f85f8d-1361-48e1-986c-9495172f4da4" (UID: "c1f85f8d-1361-48e1-986c-9495172f4da4"). InnerVolumeSpecName "kube-api-access-tmv5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:45.962923 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:45.962897    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-provider-credentials" pod "c1f85f8d-1361-48e1-986c-9495172f4da4" (UID: "c1f85f8d-1361-48e1-986c-9495172f4da4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:46.061681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.061595    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmv5r\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-kube-api-access-tmv5r\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:46.061681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.061627    4956 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1f85f8d-1361-48e1-986c-9495172f4da4-provider-credentials\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:46.083043 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.083010    4956 generic.go:356] "Generic (PLEG): container finished" podID="c1f85f8d-1361-48e1-986c-9495172f4da4" containerID="ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9" exitCode=0
Nov 02 18:26:46.083243 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.083059    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p" event={"ID":"c1f85f8d-1361-48e1-986c-9495172f4da4","Type":"ContainerDied","Data":"ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9"}
Nov 02 18:26:46.083243 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.083097    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p" event={"ID":"c1f85f8d-1361-48e1-986c-9495172f4da4","Type":"ContainerDied","Data":"6b848057e7cac9d619fdbc71fa1c137448a04e7000af0bffe690a744884e6f2a"}
Nov 02 18:26:46.083243 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.083061    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"
Nov 02 18:26:46.083243 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.083132    4956 scope.go:122] "RemoveContainer" containerID="ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9"
Nov 02 18:26:46.094715 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.094693    4956 scope.go:122] "RemoveContainer" containerID="ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9"
Nov 02 18:26:46.094997 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:46.094980    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9\": container with ID starting with ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9 not found: ID does not exist" containerID="ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9"
Nov 02 18:26:46.095045 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.095006    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9"} err="failed to get container status \"ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9\": rpc error: code = NotFound desc = could not find container \"ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9\": container with ID starting with ad35df42680f397cdfb6b5b28d444523f69333ca433486a40de87a21e314dcd9 not found: ID does not exist"
Nov 02 18:26:46.247193 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.247154    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"]
Nov 02 18:26:46.250791 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:46.250760    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rfs6p"]
Nov 02 18:26:47.865669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:47.865637    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f85f8d-1361-48e1-986c-9495172f4da4" path="/var/lib/kubelet/pods/c1f85f8d-1361-48e1-986c-9495172f4da4/volumes"
Nov 02 18:26:53.570704 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.570670    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"]
Nov 02 18:26:53.571404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.571388    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c1f85f8d-1361-48e1-986c-9495172f4da4" containerName="payload-processing"
Nov 02 18:26:53.571457 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.571405    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f85f8d-1361-48e1-986c-9495172f4da4" containerName="payload-processing"
Nov 02 18:26:53.571511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.571502    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f85f8d-1361-48e1-986c-9495172f4da4" containerName="payload-processing"
Nov 02 18:26:53.576544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.576526    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.635815 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.635783    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0fc530495c8c\""
Nov 02 18:26:53.661816 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.661776    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-routing-overlay-volume\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.662010 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.661940    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.662084 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.662048    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-plugins-config-volume\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.662317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.662292    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwsw\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-kube-api-access-nkwsw\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.687404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.687373    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"]
Nov 02 18:26:53.764252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.764223    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwsw\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-kube-api-access-nkwsw\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.764458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.764280    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-routing-overlay-volume\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.764458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.764335    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.764458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.764392    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-plugins-config-volume\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.764989 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.764963    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-routing-overlay-volume\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.765177 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.765155    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-plugins-config-volume\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.766960 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.766938    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.821857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.821783    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwsw\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-kube-api-access-nkwsw\") pod \"payload-processing-external-model-5f4b5cccd6-xhwfg\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:53.890663 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:53.890631    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:54.268784 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:54.268750    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"]
Nov 02 18:26:54.269246 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:26:54.269219    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10027426_1751_45d4_a85e_243de8bb0363.slice/crio-949aad011b9e3dfc6ee8ab7855d54329b3f574826e94880c989cae5aac4a0781 WatchSource:0}: Error finding container 949aad011b9e3dfc6ee8ab7855d54329b3f574826e94880c989cae5aac4a0781: Status 404 returned error can't find the container with id 949aad011b9e3dfc6ee8ab7855d54329b3f574826e94880c989cae5aac4a0781
Nov 02 18:26:55.148371 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:55.148332    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg" event={"ID":"10027426-1751-45d4-a85e-243de8bb0363","Type":"ContainerStarted","Data":"bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3"}
Nov 02 18:26:55.148796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:55.148383    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg" event={"ID":"10027426-1751-45d4-a85e-243de8bb0363","Type":"ContainerStarted","Data":"949aad011b9e3dfc6ee8ab7855d54329b3f574826e94880c989cae5aac4a0781"}
Nov 02 18:26:55.148796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:55.148403    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:26:55.265145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:55.265067    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg" podStartSLOduration=2.265052444 podStartE2EDuration="2.265052444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:55.263914067 +0000 UTC m=+3845.520944688" watchObservedRunningTime="2026-11-02 18:26:55.265052444 +0000 UTC m=+3845.522083065"
Nov 02 18:26:56.220863 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:56.220822    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0fc530495c8c: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:56.220863 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:56.220861    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:56.221461 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:56.220929    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials podName:10027426-1751-45d4-a85e-243de8bb0363 nodeName:}" failed. No retries permitted until 2026-11-02 18:26:56.720908873 +0000 UTC m=+3846.977939474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials") pod "payload-processing-external-model-5f4b5cccd6-xhwfg" (UID: "10027426-1751-45d4-a85e-243de8bb0363") : secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:56.727149 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:56.727091    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0fc530495c8c: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:56.727149 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:56.727137    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:56.727405 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:56.727196    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials podName:10027426-1751-45d4-a85e-243de8bb0363 nodeName:}" failed. No retries permitted until 2026-11-02 18:26:57.727181973 +0000 UTC m=+3847.984212571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials") pod "payload-processing-external-model-5f4b5cccd6-xhwfg" (UID: "10027426-1751-45d4-a85e-243de8bb0363") : secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:57.746277 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:57.746240    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0fc530495c8c: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:57.746277 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:57.746271    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:57.746875 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:57.746324    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials podName:10027426-1751-45d4-a85e-243de8bb0363 nodeName:}" failed. No retries permitted until 2026-11-02 18:26:59.746309384 +0000 UTC m=+3850.003339983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials") pod "payload-processing-external-model-5f4b5cccd6-xhwfg" (UID: "10027426-1751-45d4-a85e-243de8bb0363") : secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:58.720320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:26:58.720283    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"]
Nov 02 18:26:59.055156 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:59.055047    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:26:59.773893 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:59.773862    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0fc530495c8c: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:59.773893 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:59.773895    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:26:59.774215 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:26:59.773960    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials podName:10027426-1751-45d4-a85e-243de8bb0363 nodeName:}" failed. No retries permitted until 2026-11-02 18:27:03.773940623 +0000 UTC m=+3854.030971228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials") pod "payload-processing-external-model-5f4b5cccd6-xhwfg" (UID: "10027426-1751-45d4-a85e-243de8bb0363") : secret "agc-cred-0fc530495c8c" not found
Nov 02 18:27:03.721388 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:03.721341    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg" podUID="10027426-1751-45d4-a85e-243de8bb0363" containerName="payload-processing" containerID="cri-o://bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3" gracePeriod=25
Nov 02 18:27:03.851382 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:03.851354    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0fc530495c8c: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:27:03.851382 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:03.851383    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg: secret "agc-cred-0fc530495c8c" not found
Nov 02 18:27:03.851562 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:03.851437    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials podName:10027426-1751-45d4-a85e-243de8bb0363 nodeName:}" failed. No retries permitted until 2026-11-02 18:27:11.851423956 +0000 UTC m=+3862.108454555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials") pod "payload-processing-external-model-5f4b5cccd6-xhwfg" (UID: "10027426-1751-45d4-a85e-243de8bb0363") : secret "agc-cred-0fc530495c8c" not found
Nov 02 18:27:04.005251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.005223    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:27:04.155717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.155681    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials\") pod \"10027426-1751-45d4-a85e-243de8bb0363\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") "
Nov 02 18:27:04.155944 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.155756    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-routing-overlay-volume\") pod \"10027426-1751-45d4-a85e-243de8bb0363\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") "
Nov 02 18:27:04.156025 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.156013    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-plugins-config-volume\") pod \"10027426-1751-45d4-a85e-243de8bb0363\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") "
Nov 02 18:27:04.156340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.156310    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-routing-overlay-volume" pod "10027426-1751-45d4-a85e-243de8bb0363" (UID: "10027426-1751-45d4-a85e-243de8bb0363"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:04.156458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.156323    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-kube-api-access-nkwsw\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-kube-api-access-nkwsw\") pod \"10027426-1751-45d4-a85e-243de8bb0363\" (UID: \"10027426-1751-45d4-a85e-243de8bb0363\") "
Nov 02 18:27:04.156524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.156449    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-plugins-config-volume" pod "10027426-1751-45d4-a85e-243de8bb0363" (UID: "10027426-1751-45d4-a85e-243de8bb0363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:04.156833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.156813    4956 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-routing-overlay-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:04.156948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.156837    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10027426-1751-45d4-a85e-243de8bb0363-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:04.158566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.158535    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-kube-api-access-nkwsw" pod "10027426-1751-45d4-a85e-243de8bb0363" (UID: "10027426-1751-45d4-a85e-243de8bb0363"). InnerVolumeSpecName "kube-api-access-nkwsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:04.158729 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.158708    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials" pod "10027426-1751-45d4-a85e-243de8bb0363" (UID: "10027426-1751-45d4-a85e-243de8bb0363"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:04.209582 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.209550    4956 generic.go:356] "Generic (PLEG): container finished" podID="10027426-1751-45d4-a85e-243de8bb0363" containerID="bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3" exitCode=0
Nov 02 18:27:04.209780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.209605    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"
Nov 02 18:27:04.209780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.209619    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg" event={"ID":"10027426-1751-45d4-a85e-243de8bb0363","Type":"ContainerDied","Data":"bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3"}
Nov 02 18:27:04.209780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.209675    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg" event={"ID":"10027426-1751-45d4-a85e-243de8bb0363","Type":"ContainerDied","Data":"949aad011b9e3dfc6ee8ab7855d54329b3f574826e94880c989cae5aac4a0781"}
Nov 02 18:27:04.209780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.209703    4956 scope.go:122] "RemoveContainer" containerID="bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3"
Nov 02 18:27:04.221959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.221937    4956 scope.go:122] "RemoveContainer" containerID="bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3"
Nov 02 18:27:04.222342 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:04.222322    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3\": container with ID starting with bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3 not found: ID does not exist" containerID="bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3"
Nov 02 18:27:04.222456 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.222351    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3"} err="failed to get container status \"bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3\": rpc error: code = NotFound desc = could not find container \"bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3\": container with ID starting with bc03330bab565d904a158a7b5041892819df07d43c2a25aa8c68a91268850bc3 not found: ID does not exist"
Nov 02 18:27:04.257688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.257603    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkwsw\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-kube-api-access-nkwsw\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:04.257688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.257634    4956 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10027426-1751-45d4-a85e-243de8bb0363-provider-credentials\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:04.385930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.385895    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"]
Nov 02 18:27:04.397554 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:04.397519    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-xhwfg"]
Nov 02 18:27:05.868759 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:05.868720    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10027426-1751-45d4-a85e-243de8bb0363" path="/var/lib/kubelet/pods/10027426-1751-45d4-a85e-243de8bb0363/volumes"
Nov 02 18:27:06.545005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.544969    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"]
Nov 02 18:27:06.545730 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.545705    4956 cpu_manager.go:414] "containerMap: removing container" podUID="10027426-1751-45d4-a85e-243de8bb0363" containerName="payload-processing"
Nov 02 18:27:06.545833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.545730    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10027426-1751-45d4-a85e-243de8bb0363" containerName="payload-processing"
Nov 02 18:27:06.545963 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.545949    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="10027426-1751-45d4-a85e-243de8bb0363" containerName="payload-processing"
Nov 02 18:27:06.552517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.552483    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.606941 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.606904    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"]
Nov 02 18:27:06.689521 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.689480    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.689736 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.689619    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7cb\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-kube-api-access-tn7cb\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.689736 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.689695    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.689830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.689776    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.790635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.790594    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.790830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.790695    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7cb\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-kube-api-access-tn7cb\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.790830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.790731    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.790830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.790777    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.791515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.791487    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.791632 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.791526    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.793297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.793280    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.847892 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.847812    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7cb\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-kube-api-access-tn7cb\") pod \"payload-processing-external-model-596fb6c5c4-69s4x\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:06.869897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:06.869859    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:07.262135 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:07.262096    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"]
Nov 02 18:27:07.263518 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:27:07.263493    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07985da9_31f1_4365_8a90_9be674b0e6ab.slice/crio-6d4d0728c5b5de4e46fc5a030f6b5937662a6951a1156b4b8a27cf67e4488466 WatchSource:0}: Error finding container 6d4d0728c5b5de4e46fc5a030f6b5937662a6951a1156b4b8a27cf67e4488466: Status 404 returned error can't find the container with id 6d4d0728c5b5de4e46fc5a030f6b5937662a6951a1156b4b8a27cf67e4488466
Nov 02 18:27:08.244219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:08.244171    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" event={"ID":"07985da9-31f1-4365-8a90-9be674b0e6ab","Type":"ContainerStarted","Data":"68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644"}
Nov 02 18:27:08.244732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:08.244237    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" event={"ID":"07985da9-31f1-4365-8a90-9be674b0e6ab","Type":"ContainerStarted","Data":"6d4d0728c5b5de4e46fc5a030f6b5937662a6951a1156b4b8a27cf67e4488466"}
Nov 02 18:27:08.244732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:08.244371    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:27:08.362433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:08.362368    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podStartSLOduration=2.362352328 podStartE2EDuration="2.362352328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:08.360787542 +0000 UTC m=+3858.617818164" watchObservedRunningTime="2026-11-02 18:27:08.362352328 +0000 UTC m=+3858.619382947"
Nov 02 18:27:17.302192 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.302154    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"]
Nov 02 18:27:17.307883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.307861    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:17.332048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.332015    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321bd38a-f405-47e5-b9a7-52c6fe4df243-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:17.332233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.332207    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cgd\" (UniqueName: \"kubernetes.io/projected/321bd38a-f405-47e5-b9a7-52c6fe4df243-kube-api-access-v9cgd\") pod \"payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:17.361415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.361386    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"]
Nov 02 18:27:17.407001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.406939    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-662a80\""
Nov 02 18:27:17.407225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.407036    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-662a80-dockercfg-7xgdx\""
Nov 02 18:27:17.433403 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.433365    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cgd\" (UniqueName: \"kubernetes.io/projected/321bd38a-f405-47e5-b9a7-52c6fe4df243-kube-api-access-v9cgd\") pod \"payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:17.433561 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.433535    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321bd38a-f405-47e5-b9a7-52c6fe4df243-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:17.434254 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.434236    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321bd38a-f405-47e5-b9a7-52c6fe4df243-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:17.490415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.490378    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cgd\" (UniqueName: \"kubernetes.io/projected/321bd38a-f405-47e5-b9a7-52c6fe4df243-kube-api-access-v9cgd\") pod \"payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:17.624086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:17.623983    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:18.002757 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:18.002723    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"]
Nov 02 18:27:18.003617 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:27:18.003582    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321bd38a_f405_47e5_b9a7_52c6fe4df243.slice/crio-b3883ccd1f41497f9b12df6bff3321a5c852ea7e3decf8400588f1829dbdc1bd WatchSource:0}: Error finding container b3883ccd1f41497f9b12df6bff3321a5c852ea7e3decf8400588f1829dbdc1bd: Status 404 returned error can't find the container with id b3883ccd1f41497f9b12df6bff3321a5c852ea7e3decf8400588f1829dbdc1bd
Nov 02 18:27:18.312315 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:18.312215    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz" event={"ID":"321bd38a-f405-47e5-b9a7-52c6fe4df243","Type":"ContainerStarted","Data":"66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9"}
Nov 02 18:27:18.312701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:18.312311    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz" event={"ID":"321bd38a-f405-47e5-b9a7-52c6fe4df243","Type":"ContainerStarted","Data":"b3883ccd1f41497f9b12df6bff3321a5c852ea7e3decf8400588f1829dbdc1bd"}
Nov 02 18:27:18.312701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:18.312456    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:18.428054 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:18.427998    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz" podStartSLOduration=1.427982773 podStartE2EDuration="1.427982773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:18.425578881 +0000 UTC m=+3868.682609501" watchObservedRunningTime="2026-11-02 18:27:18.427982773 +0000 UTC m=+3868.685013387"
Nov 02 18:27:19.254526 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:19.254484    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:27:21.446402 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:21.446356    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"]
Nov 02 18:27:25.972426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:25.972380    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"]
Nov 02 18:27:25.976680 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:25.976662    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.059485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.059453    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9631198f-23f9-4749-98e2-84573f52b251-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.059670 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.059520    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9631198f-23f9-4749-98e2-84573f52b251-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.059670 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.059633    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.059788 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.059671    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.059845 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.059782    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw259\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-kube-api-access-gw259\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.059900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.059869    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.060014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.059996    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.060063 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.060051    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.060149 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.060103    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161174    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161219    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161335    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9631198f-23f9-4749-98e2-84573f52b251-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161518 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161396    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9631198f-23f9-4749-98e2-84573f52b251-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161518 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161440    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161518 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161492    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161535    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw259\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-kube-api-access-gw259\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161672    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.161890 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161862    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.162042 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.162017    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.162189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.161958    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.162189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.162109    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9631198f-23f9-4749-98e2-84573f52b251-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.162508 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.162198    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.162508 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.162329    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.163667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.163643    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.164132 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.164098    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9631198f-23f9-4749-98e2-84573f52b251-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.184042 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.184007    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"]
Nov 02 18:27:26.223564 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.223476    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw259\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-kube-api-access-gw259\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.225788 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.225755    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-lkfz5\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.291825 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.291796    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:26.447353 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.447312    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4" podUID="4bb23074-96da-4827-b1fb-9f344e15b68a" containerName="payload-processing" containerID="cri-o://90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be" gracePeriod=25
Nov 02 18:27:26.702101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.702074    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:27:26.736782 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.736750    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"]
Nov 02 18:27:26.737620 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:27:26.737593    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9631198f_23f9_4749_98e2_84573f52b251.slice/crio-0bef2c7befa20b05a267f0ee6b0e233799627fb73416fbf3edb02755183386e8 WatchSource:0}: Error finding container 0bef2c7befa20b05a267f0ee6b0e233799627fb73416fbf3edb02755183386e8: Status 404 returned error can't find the container with id 0bef2c7befa20b05a267f0ee6b0e233799627fb73416fbf3edb02755183386e8
Nov 02 18:27:26.740525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.740495    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:27:26.740615 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.740561    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:27:26.740615 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.740592    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:27:26.776306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.776277    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb23074-96da-4827-b1fb-9f344e15b68a-kube-api-access-2jhgg\" (UniqueName: \"kubernetes.io/projected/4bb23074-96da-4827-b1fb-9f344e15b68a-kube-api-access-2jhgg\") pod \"4bb23074-96da-4827-b1fb-9f344e15b68a\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") "
Nov 02 18:27:26.776507 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.776491    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb23074-96da-4827-b1fb-9f344e15b68a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb23074-96da-4827-b1fb-9f344e15b68a-plugins-config-volume\") pod \"4bb23074-96da-4827-b1fb-9f344e15b68a\" (UID: \"4bb23074-96da-4827-b1fb-9f344e15b68a\") "
Nov 02 18:27:26.776898 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.776869    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb23074-96da-4827-b1fb-9f344e15b68a-plugins-config-volume" pod "4bb23074-96da-4827-b1fb-9f344e15b68a" (UID: "4bb23074-96da-4827-b1fb-9f344e15b68a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:26.777071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.777056    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb23074-96da-4827-b1fb-9f344e15b68a-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:26.778805 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.778779    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb23074-96da-4827-b1fb-9f344e15b68a-kube-api-access-2jhgg" pod "4bb23074-96da-4827-b1fb-9f344e15b68a" (UID: "4bb23074-96da-4827-b1fb-9f344e15b68a"). InnerVolumeSpecName "kube-api-access-2jhgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:26.878251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:26.878216    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jhgg\" (UniqueName: \"kubernetes.io/projected/4bb23074-96da-4827-b1fb-9f344e15b68a-kube-api-access-2jhgg\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:27.389449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.389411    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" event={"ID":"9631198f-23f9-4749-98e2-84573f52b251","Type":"ContainerStarted","Data":"40ce381ee443d9dc815a4cfa6eefbd17b229bc8fdabe6aa0be00d3f01a56b66b"}
Nov 02 18:27:27.389934 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.389484    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" event={"ID":"9631198f-23f9-4749-98e2-84573f52b251","Type":"ContainerStarted","Data":"0bef2c7befa20b05a267f0ee6b0e233799627fb73416fbf3edb02755183386e8"}
Nov 02 18:27:27.390904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.390877    4956 generic.go:356] "Generic (PLEG): container finished" podID="4bb23074-96da-4827-b1fb-9f344e15b68a" containerID="90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be" exitCode=0
Nov 02 18:27:27.391024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.390914    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"
Nov 02 18:27:27.391024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.390959    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4" event={"ID":"4bb23074-96da-4827-b1fb-9f344e15b68a","Type":"ContainerDied","Data":"90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be"}
Nov 02 18:27:27.391024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.391016    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4" event={"ID":"4bb23074-96da-4827-b1fb-9f344e15b68a","Type":"ContainerDied","Data":"bdc480f98d91dd4d9024e25a9603481186d5d515fe529641cc7b5aaca0341e72"}
Nov 02 18:27:27.391173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.391043    4956 scope.go:122] "RemoveContainer" containerID="90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be"
Nov 02 18:27:27.402973 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.402555    4956 scope.go:122] "RemoveContainer" containerID="90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be"
Nov 02 18:27:27.403073 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:27.402994    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be\": container with ID starting with 90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be not found: ID does not exist" containerID="90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be"
Nov 02 18:27:27.403073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.403033    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be"} err="failed to get container status \"90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be\": rpc error: code = NotFound desc = could not find container \"90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be\": container with ID starting with 90046ae9ecb73090822248ebdd02ae5f31683964c1fedac5f3dcf34e2ae1b8be not found: ID does not exist"
Nov 02 18:27:27.615472 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.615415    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podStartSLOduration=2.615400464 podStartE2EDuration="2.615400464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:27.564334877 +0000 UTC m=+3877.821365496" watchObservedRunningTime="2026-11-02 18:27:27.615400464 +0000 UTC m=+3877.872431084"
Nov 02 18:27:27.683424 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.683376    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"]
Nov 02 18:27:27.686890 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.686864    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e4b90c-584cc4b4b7-zdkf4"]
Nov 02 18:27:27.867071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:27.866989    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb23074-96da-4827-b1fb-9f344e15b68a" path="/var/lib/kubelet/pods/4bb23074-96da-4827-b1fb-9f344e15b68a/volumes"
Nov 02 18:27:28.282978 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.282513    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"]
Nov 02 18:27:28.283326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.283104    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy" containerID="cri-o://529dac07f55c5cf14875d9c8d72ebeaa1fc149a36a2bf374665d5b6dbc0fa47a" gracePeriod=30
Nov 02 18:27:28.292234 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.292210    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:28.293886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.293861    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 18:27:28.293994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.293917    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 18:27:28.311633 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.311604    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"]
Nov 02 18:27:28.311873 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.311843    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy" containerID="cri-o://c22c4b8cb9859d1c2a716063d5f1e4c799cb97b695d4570ff50fad39d6d2e855" gracePeriod=30
Nov 02 18:27:28.314816 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.314789    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"]
Nov 02 18:27:28.314969 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:28.314952    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy" containerID="cri-o://8e0e7fa529d52611f26a3fff4f09f0fc0f0492d88bb0ba4a1908e64e16af81be" gracePeriod=30
Nov 02 18:27:29.077772 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:29.077733    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:27:29.260478 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:29.258563    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:27:29.293110 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:29.293076    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 18:27:29.293304 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:29.293156    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 18:27:29.321988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:29.321958    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:27:30.292735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:30.292696    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 18:27:30.293235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:30.292767    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 18:27:31.292335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:31.292299    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 18:27:31.292547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:31.292365    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 18:27:32.293088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:32.293045    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 18:27:32.293620 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:32.293151    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 18:27:33.293262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.293226    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 18:27:33.293670 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.293288    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 18:27:33.468464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.468257    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" event={"ID":"3f7d4c4c-2363-4cd5-903b-10832fe15b8a","Type":"ContainerDied","Data":"529dac07f55c5cf14875d9c8d72ebeaa1fc149a36a2bf374665d5b6dbc0fa47a"}
Nov 02 18:27:33.468464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.468348    4956 scope.go:122] "RemoveContainer" containerID="1fab816052b0b3334280ac04544fb3de250d859b353d5595f39d2e4983bd0354"
Nov 02 18:27:33.470336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.468111    4956 generic.go:356] "Generic (PLEG): container finished" podID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerID="529dac07f55c5cf14875d9c8d72ebeaa1fc149a36a2bf374665d5b6dbc0fa47a" exitCode=0
Nov 02 18:27:33.472590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.472420    4956 generic.go:356] "Generic (PLEG): container finished" podID="75b1abdf-ba68-4f11-a240-93914229abcd" containerID="8e0e7fa529d52611f26a3fff4f09f0fc0f0492d88bb0ba4a1908e64e16af81be" exitCode=0
Nov 02 18:27:33.472590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.472510    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" event={"ID":"75b1abdf-ba68-4f11-a240-93914229abcd","Type":"ContainerDied","Data":"8e0e7fa529d52611f26a3fff4f09f0fc0f0492d88bb0ba4a1908e64e16af81be"}
Nov 02 18:27:33.475495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.475270    4956 generic.go:356] "Generic (PLEG): container finished" podID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerID="c22c4b8cb9859d1c2a716063d5f1e4c799cb97b695d4570ff50fad39d6d2e855" exitCode=0
Nov 02 18:27:33.475495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.475426    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" event={"ID":"af46991d-98e5-4c24-a847-17a0a8b3b07a","Type":"ContainerDied","Data":"c22c4b8cb9859d1c2a716063d5f1e4c799cb97b695d4570ff50fad39d6d2e855"}
Nov 02 18:27:33.621609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.621569    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:27:33.682167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.682139    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:27:33.685711 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.685687    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:27:33.795206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795168    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-certs\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.795420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795222    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-envoy\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.795420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795253    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-kube-api-access-szsch\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-kube-api-access-szsch\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.795420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795286    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istiod-ca-cert\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.795420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795355    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75b1abdf-ba68-4f11-a240-93914229abcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b1abdf-ba68-4f11-a240-93914229abcd-istio-podinfo\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.795420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795394    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-data\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795442    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-envoy\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795490    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af46991d-98e5-4c24-a847-17a0a8b3b07a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af46991d-98e5-4c24-a847-17a0a8b3b07a-istiod-ca-cert\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795540    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-credential-socket\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795598    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-credential-socket\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795633    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-podinfo\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795619    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-certs" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795663    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-envoy\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.795716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795689    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b1abdf-ba68-4f11-a240-93914229abcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b1abdf-ba68-4f11-a240-93914229abcd-istiod-ca-cert\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795722    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-data\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795778    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-kube-api-access-xnv82\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-kube-api-access-xnv82\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795829    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-socket\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795852    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-data" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795865    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-socket\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795912    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-token\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-token\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795955    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-data\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.795987    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-kube-api-access-dqs2g\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-kube-api-access-dqs2g\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796024    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-certs\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796055    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-podinfo\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.796145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796094    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-certs\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.796758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796179    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-socket\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.796758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796232    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-istio-token\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-istio-token\") pod \"75b1abdf-ba68-4f11-a240-93914229abcd\" (UID: \"75b1abdf-ba68-4f11-a240-93914229abcd\") "
Nov 02 18:27:33.796758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796260    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-token\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-token\") pod \"af46991d-98e5-4c24-a847-17a0a8b3b07a\" (UID: \"af46991d-98e5-4c24-a847-17a0a8b3b07a\") "
Nov 02 18:27:33.796758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796322    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-credential-socket\") pod \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\" (UID: \"3f7d4c4c-2363-4cd5-903b-10832fe15b8a\") "
Nov 02 18:27:33.796758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796632    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istiod-ca-cert" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:33.797017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.796899    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-credential-socket" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.797356 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.797328    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.797474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.797363    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.797474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.797378    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.797474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.797395    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.798658 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.798628    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b1abdf-ba68-4f11-a240-93914229abcd-istiod-ca-cert" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:33.798877 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.798853    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-data" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.799168 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.799144    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af46991d-98e5-4c24-a847-17a0a8b3b07a-istiod-ca-cert" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:33.799403 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.799380    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-credential-socket" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.799735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.799689    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-credential-socket" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.800001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.799977    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-socket" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.800251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.800230    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-socket" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.800354 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.800337    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-kube-api-access-szsch" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "kube-api-access-szsch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.800540 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.800524    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-socket" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.800772 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.800748    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-certs" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.801543 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.801505    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-data" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.801874 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.801707    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-certs" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.802056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.802020    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-envoy" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.804739 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.804701    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b1abdf-ba68-4f11-a240-93914229abcd-istio-podinfo" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:33.806240 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.806189    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-kube-api-access-dqs2g" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "kube-api-access-dqs2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.806832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.806567    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-podinfo" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:33.806832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.806617    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-envoy" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.806832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.806633    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-istio-token" pod "75b1abdf-ba68-4f11-a240-93914229abcd" (UID: "75b1abdf-ba68-4f11-a240-93914229abcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.806832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.806702    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-podinfo" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:33.806832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.806808    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-envoy" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.807363 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.807331    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-token" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.807463 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.807428    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-kube-api-access-xnv82" pod "3f7d4c4c-2363-4cd5-903b-10832fe15b8a" (UID: "3f7d4c4c-2363-4cd5-903b-10832fe15b8a"). InnerVolumeSpecName "kube-api-access-xnv82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.808437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.808412    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-token" pod "af46991d-98e5-4c24-a847-17a0a8b3b07a" (UID: "af46991d-98e5-4c24-a847-17a0a8b3b07a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.898926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898899    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.898926 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898926    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898937    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898946    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898955    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqs2g\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-kube-api-access-dqs2g\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898964    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898973    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898981    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898990    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.898998    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b1abdf-ba68-4f11-a240-93914229abcd-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899006    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899014    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899023    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szsch\" (UniqueName: \"kubernetes.io/projected/af46991d-98e5-4c24-a847-17a0a8b3b07a-kube-api-access-szsch\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899033    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b1abdf-ba68-4f11-a240-93914229abcd-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899041    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899050    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af46991d-98e5-4c24-a847-17a0a8b3b07a-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899058    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af46991d-98e5-4c24-a847-17a0a8b3b07a-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899068    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899077    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899085    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899093    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b1abdf-ba68-4f11-a240-93914229abcd-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899102    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b1abdf-ba68-4f11-a240-93914229abcd-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.899695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:33.899110    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnv82\" (UniqueName: \"kubernetes.io/projected/3f7d4c4c-2363-4cd5-903b-10832fe15b8a-kube-api-access-xnv82\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.292222 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.292180    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 18:27:34.292414 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.292252    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 18:27:34.483810 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.483774    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"
Nov 02 18:27:34.484334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.483808    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z" event={"ID":"3f7d4c4c-2363-4cd5-903b-10832fe15b8a","Type":"ContainerDied","Data":"8875d8e6bf27325d59b7889045e0c44437bf66793e69ac76f737ea2cce360f2a"}
Nov 02 18:27:34.484334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.483878    4956 scope.go:122] "RemoveContainer" containerID="529dac07f55c5cf14875d9c8d72ebeaa1fc149a36a2bf374665d5b6dbc0fa47a"
Nov 02 18:27:34.485798 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.485551    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"
Nov 02 18:27:34.486281 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.485958    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx" event={"ID":"75b1abdf-ba68-4f11-a240-93914229abcd","Type":"ContainerDied","Data":"06973fd9f06deece9486ad477a48dec924266d34d5ac9dddecc7d7d7fae4f4d9"}
Nov 02 18:27:34.487937 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.487911    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b" event={"ID":"af46991d-98e5-4c24-a847-17a0a8b3b07a","Type":"ContainerDied","Data":"3a9318f136897ae1164b825140f36bb798d61baea9d32a2c1ff38b05245a3fcc"}
Nov 02 18:27:34.488065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.487966    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"
Nov 02 18:27:34.499023 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.499002    4956 scope.go:122] "RemoveContainer" containerID="8e0e7fa529d52611f26a3fff4f09f0fc0f0492d88bb0ba4a1908e64e16af81be"
Nov 02 18:27:34.509525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.509468    4956 scope.go:122] "RemoveContainer" containerID="c22c4b8cb9859d1c2a716063d5f1e4c799cb97b695d4570ff50fad39d6d2e855"
Nov 02 18:27:34.648461 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.648363    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"]
Nov 02 18:27:34.655056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.655025    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-9p4fx"]
Nov 02 18:27:34.777990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.777952    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"]
Nov 02 18:27:34.789818 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.789786    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-jqf2z"]
Nov 02 18:27:34.898816 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.898722    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"]
Nov 02 18:27:34.908489 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:34.908456    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6qs8b"]
Nov 02 18:27:35.583537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:35.583496    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:35.584028 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:35.584012    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:35.584864 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:35.584831    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:27:35.866719 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:35.866633    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" path="/var/lib/kubelet/pods/3f7d4c4c-2363-4cd5-903b-10832fe15b8a/volumes"
Nov 02 18:27:35.867323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:35.867300    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" path="/var/lib/kubelet/pods/75b1abdf-ba68-4f11-a240-93914229abcd/volumes"
Nov 02 18:27:35.868198 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:35.868173    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" path="/var/lib/kubelet/pods/af46991d-98e5-4c24-a847-17a0a8b3b07a/volumes"
Nov 02 18:27:37.631753 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.631718    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"]
Nov 02 18:27:37.632522 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632501    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy"
Nov 02 18:27:37.632614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632524    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy"
Nov 02 18:27:37.632614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632539    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4bb23074-96da-4827-b1fb-9f344e15b68a" containerName="payload-processing"
Nov 02 18:27:37.632614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632545    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb23074-96da-4827-b1fb-9f344e15b68a" containerName="payload-processing"
Nov 02 18:27:37.632614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632559    4956 cpu_manager.go:414] "containerMap: removing container" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy"
Nov 02 18:27:37.632614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632566    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy"
Nov 02 18:27:37.632614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632594    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy"
Nov 02 18:27:37.632614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632602    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy"
Nov 02 18:27:37.632823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632623    4956 cpu_manager.go:414] "containerMap: removing container" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy"
Nov 02 18:27:37.632823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632628    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy"
Nov 02 18:27:37.632823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632756    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy"
Nov 02 18:27:37.632823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632770    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="af46991d-98e5-4c24-a847-17a0a8b3b07a" containerName="istio-proxy"
Nov 02 18:27:37.632823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632779    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb23074-96da-4827-b1fb-9f344e15b68a" containerName="payload-processing"
Nov 02 18:27:37.632823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.632786    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b1abdf-ba68-4f11-a240-93914229abcd" containerName="istio-proxy"
Nov 02 18:27:37.637787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.637769    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:37.692562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.692523    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"]
Nov 02 18:27:37.738130 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.738093    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-546881\""
Nov 02 18:27:37.746988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.746954    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hqb\" (UniqueName: \"kubernetes.io/projected/b0ca09da-36ff-4a0f-b87b-948bb48eb768-kube-api-access-z2hqb\") pod \"payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:37.746988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.747004    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ca09da-36ff-4a0f-b87b-948bb48eb768-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:37.780716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.780676    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"]
Nov 02 18:27:37.782041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.782006    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7d4c4c-2363-4cd5-903b-10832fe15b8a" containerName="istio-proxy"
Nov 02 18:27:37.790159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.786515    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:37.839152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.839097    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-546881-dockercfg-5d864\""
Nov 02 18:27:37.841137 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.841099    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"]
Nov 02 18:27:37.848172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.848136    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hqb\" (UniqueName: \"kubernetes.io/projected/b0ca09da-36ff-4a0f-b87b-948bb48eb768-kube-api-access-z2hqb\") pod \"payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:37.848460 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.848384    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ca09da-36ff-4a0f-b87b-948bb48eb768-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:37.848677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.848653    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkp6r\" (UniqueName: \"kubernetes.io/projected/9e1212eb-9fce-4603-b954-32c1b28fbc6a-kube-api-access-jkp6r\") pod \"payload-processing-e2e-inf-a-546881-849d945fc-2m9vm\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:37.848873 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.848843    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1212eb-9fce-4603-b954-32c1b28fbc6a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-546881-849d945fc-2m9vm\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:37.850215 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.850184    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ca09da-36ff-4a0f-b87b-948bb48eb768-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:37.906942 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.906864    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hqb\" (UniqueName: \"kubernetes.io/projected/b0ca09da-36ff-4a0f-b87b-948bb48eb768-kube-api-access-z2hqb\") pod \"payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:37.949981 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.949938    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkp6r\" (UniqueName: \"kubernetes.io/projected/9e1212eb-9fce-4603-b954-32c1b28fbc6a-kube-api-access-jkp6r\") pod \"payload-processing-e2e-inf-a-546881-849d945fc-2m9vm\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:37.950214 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.950018    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1212eb-9fce-4603-b954-32c1b28fbc6a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-546881-849d945fc-2m9vm\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:37.950836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.950812    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1212eb-9fce-4603-b954-32c1b28fbc6a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-546881-849d945fc-2m9vm\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:37.952669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:37.952650    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:38.008735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.008697    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkp6r\" (UniqueName: \"kubernetes.io/projected/9e1212eb-9fce-4603-b954-32c1b28fbc6a-kube-api-access-jkp6r\") pod \"payload-processing-e2e-inf-a-546881-849d945fc-2m9vm\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:38.104826 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.104791    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:38.332299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.332268    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"]
Nov 02 18:27:38.333194 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:27:38.333101    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ca09da_36ff_4a0f_b87b_948bb48eb768.slice/crio-a7bb063d9f51c377b04ec2bd32256f49c866c4cb4773f2419e8c1e9a6339cd45 WatchSource:0}: Error finding container a7bb063d9f51c377b04ec2bd32256f49c866c4cb4773f2419e8c1e9a6339cd45: Status 404 returned error can't find the container with id a7bb063d9f51c377b04ec2bd32256f49c866c4cb4773f2419e8c1e9a6339cd45
Nov 02 18:27:38.487822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.487791    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"]
Nov 02 18:27:38.488523 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:27:38.488498    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1212eb_9fce_4603_b954_32c1b28fbc6a.slice/crio-06bb80a440b38cc3fd7788acc6052afbde449c28ae498feef784a6cb7a1bfbd9 WatchSource:0}: Error finding container 06bb80a440b38cc3fd7788acc6052afbde449c28ae498feef784a6cb7a1bfbd9: Status 404 returned error can't find the container with id 06bb80a440b38cc3fd7788acc6052afbde449c28ae498feef784a6cb7a1bfbd9
Nov 02 18:27:38.524001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.523967    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h" event={"ID":"b0ca09da-36ff-4a0f-b87b-948bb48eb768","Type":"ContainerStarted","Data":"b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e"}
Nov 02 18:27:38.524177 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.524032    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h" event={"ID":"b0ca09da-36ff-4a0f-b87b-948bb48eb768","Type":"ContainerStarted","Data":"a7bb063d9f51c377b04ec2bd32256f49c866c4cb4773f2419e8c1e9a6339cd45"}
Nov 02 18:27:38.524177 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.524148    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:38.525315 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.525288    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm" event={"ID":"9e1212eb-9fce-4603-b954-32c1b28fbc6a","Type":"ContainerStarted","Data":"06bb80a440b38cc3fd7788acc6052afbde449c28ae498feef784a6cb7a1bfbd9"}
Nov 02 18:27:38.641483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:38.641426    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h" podStartSLOduration=1.641410464 podStartE2EDuration="1.641410464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:38.639695653 +0000 UTC m=+3888.896726273" watchObservedRunningTime="2026-11-02 18:27:38.641410464 +0000 UTC m=+3888.898441083"
Nov 02 18:27:39.252289 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.252248    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:27:39.463535 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.463501    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"]
Nov 02 18:27:39.463723 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.463702    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f" podUID="b0a914c3-1d52-4610-a7cb-d3d50ec65664" containerName="maas-api" containerID="cri-o://6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6" gracePeriod=30
Nov 02 18:27:39.536289 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.536193    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm" event={"ID":"9e1212eb-9fce-4603-b954-32c1b28fbc6a","Type":"ContainerStarted","Data":"dbe2b4e943857f1e8b0955bc2eb32c705cc4c6f3555c0dff1fe2cd70516f28e7"}
Nov 02 18:27:39.536464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.536431    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:39.695205 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.695143    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm" podStartSLOduration=2.695098502 podStartE2EDuration="2.695098502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:39.693840471 +0000 UTC m=+3889.950871092" watchObservedRunningTime="2026-11-02 18:27:39.695098502 +0000 UTC m=+3889.952129122"
Nov 02 18:27:39.740512 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.740483    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:27:39.775521 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.775489    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls\") pod \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") "
Nov 02 18:27:39.775711 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.775560    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a914c3-1d52-4610-a7cb-d3d50ec65664-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a914c3-1d52-4610-a7cb-d3d50ec65664-openshift-service-ca\") pod \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") "
Nov 02 18:27:39.775711 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.775630    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-metrics-certs\") pod \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") "
Nov 02 18:27:39.775711 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.775697    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a914c3-1d52-4610-a7cb-d3d50ec65664-kube-api-access-w862d\" (UniqueName: \"kubernetes.io/projected/b0a914c3-1d52-4610-a7cb-d3d50ec65664-kube-api-access-w862d\") pod \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\" (UID: \"b0a914c3-1d52-4610-a7cb-d3d50ec65664\") "
Nov 02 18:27:39.776003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.775965    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a914c3-1d52-4610-a7cb-d3d50ec65664-openshift-service-ca" pod "b0a914c3-1d52-4610-a7cb-d3d50ec65664" (UID: "b0a914c3-1d52-4610-a7cb-d3d50ec65664"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:39.777960 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.777930    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-metrics-certs" pod "b0a914c3-1d52-4610-a7cb-d3d50ec65664" (UID: "b0a914c3-1d52-4610-a7cb-d3d50ec65664"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:39.778067 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.777997    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls" pod "b0a914c3-1d52-4610-a7cb-d3d50ec65664" (UID: "b0a914c3-1d52-4610-a7cb-d3d50ec65664"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:39.778162 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.778101    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a914c3-1d52-4610-a7cb-d3d50ec65664-kube-api-access-w862d" pod "b0a914c3-1d52-4610-a7cb-d3d50ec65664" (UID: "b0a914c3-1d52-4610-a7cb-d3d50ec65664"). InnerVolumeSpecName "kube-api-access-w862d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:39.877177 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.877142    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w862d\" (UniqueName: \"kubernetes.io/projected/b0a914c3-1d52-4610-a7cb-d3d50ec65664-kube-api-access-w862d\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:39.877177 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.877178    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:39.877480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.877196    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a914c3-1d52-4610-a7cb-d3d50ec65664-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:39.877480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:39.877243    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a914c3-1d52-4610-a7cb-d3d50ec65664-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:40.544488 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.544457    4956 generic.go:356] "Generic (PLEG): container finished" podID="b0a914c3-1d52-4610-a7cb-d3d50ec65664" containerID="6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6" exitCode=0
Nov 02 18:27:40.544717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.544505    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"
Nov 02 18:27:40.544717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.544542    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f" event={"ID":"b0a914c3-1d52-4610-a7cb-d3d50ec65664","Type":"ContainerDied","Data":"6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6"}
Nov 02 18:27:40.544717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.544609    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f" event={"ID":"b0a914c3-1d52-4610-a7cb-d3d50ec65664","Type":"ContainerDied","Data":"0d2369dc970b88017a5cfd262c7803309ad4d60c80d7319655c7c9fb2240b3e4"}
Nov 02 18:27:40.544717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.544639    4956 scope.go:122] "RemoveContainer" containerID="6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6"
Nov 02 18:27:40.554950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.554929    4956 scope.go:122] "RemoveContainer" containerID="6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6"
Nov 02 18:27:40.555274 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:40.555253    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6\": container with ID starting with 6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6 not found: ID does not exist" containerID="6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6"
Nov 02 18:27:40.555339 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.555292    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6"} err="failed to get container status \"6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6\": rpc error: code = NotFound desc = could not find container \"6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6\": container with ID starting with 6e1d335f4a8fc975d846057b7e51fcd7cd40faafa47859b10292e05ae118acd6 not found: ID does not exist"
Nov 02 18:27:40.710022 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.709983    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"]
Nov 02 18:27:40.759853 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:40.759819    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a9496c-6c66cb6cc-9rl9f"]
Nov 02 18:27:41.081578 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.081534    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"]
Nov 02 18:27:41.082531 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.082507    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b0a914c3-1d52-4610-a7cb-d3d50ec65664" containerName="maas-api"
Nov 02 18:27:41.082531 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.082528    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a914c3-1d52-4610-a7cb-d3d50ec65664" containerName="maas-api"
Nov 02 18:27:41.082760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.082658    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a914c3-1d52-4610-a7cb-d3d50ec65664" containerName="maas-api"
Nov 02 18:27:41.088060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.088031    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.186052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.186017    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"]
Nov 02 18:27:41.194244 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194213    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194321    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194376    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194407    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vwcd\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-kube-api-access-5vwcd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194443    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194536    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194567    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194598    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.194948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.194642    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ed5b33-070f-49e9-a5c2-635c0b220827-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.295611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.295578    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.295795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.295620    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.295795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.295735    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ed5b33-070f-49e9-a5c2-635c0b220827-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.295946 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.295925    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.295958    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.295963    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296047    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296072    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vwcd\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-kube-api-access-5vwcd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296105    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296207    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296246    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296455 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296405    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296507 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296488    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.296601 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.296576    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ed5b33-070f-49e9-a5c2-635c0b220827-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.298399 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.298366    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.298805 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.298784    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.357066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.356988    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vwcd\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-kube-api-access-5vwcd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.357456 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.357432    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-prpkv\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.405835 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.405802    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:41.840053 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.840016    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"]
Nov 02 18:27:41.845562 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:27:41.845519    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ed5b33_070f_49e9_a5c2_635c0b220827.slice/crio-770afe19f03599f8cc27ddbee3512610d610b624061a50c3fb5a88eb7ebf8940 WatchSource:0}: Error finding container 770afe19f03599f8cc27ddbee3512610d610b624061a50c3fb5a88eb7ebf8940: Status 404 returned error can't find the container with id 770afe19f03599f8cc27ddbee3512610d610b624061a50c3fb5a88eb7ebf8940
Nov 02 18:27:41.848491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.848455    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:27:41.848608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.848535    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:27:41.848608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.848574    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:27:41.868215 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:41.868179    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a914c3-1d52-4610-a7cb-d3d50ec65664" path="/var/lib/kubelet/pods/b0a914c3-1d52-4610-a7cb-d3d50ec65664/volumes"
Nov 02 18:27:41.904902 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:41.904860    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:41.904902 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:41.904891    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:41.905217 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:41.904944    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:27:42.40492573 +0000 UTC m=+3892.661956344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:42.412928 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:42.412897    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:42.412928 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:42.412923    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:42.413200 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:42.412981    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:27:43.412966328 +0000 UTC m=+3893.669996939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:42.562531 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:42.562492    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" event={"ID":"78ed5b33-070f-49e9-a5c2-635c0b220827","Type":"ContainerStarted","Data":"29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481"}
Nov 02 18:27:42.562712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:42.562550    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" event={"ID":"78ed5b33-070f-49e9-a5c2-635c0b220827","Type":"ContainerStarted","Data":"770afe19f03599f8cc27ddbee3512610d610b624061a50c3fb5a88eb7ebf8940"}
Nov 02 18:27:42.682560 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:42.682508    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podStartSLOduration=1.682492301 podStartE2EDuration="1.682492301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:42.679377905 +0000 UTC m=+3892.936408525" watchObservedRunningTime="2026-11-02 18:27:42.682492301 +0000 UTC m=+3892.939522920"
Nov 02 18:27:43.406940 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:43.406902    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:43.408463 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:43.408436    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-prpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:27:43.408589 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:43.408487    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 02 18:27:43.433157 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:43.431402    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:43.433157 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:43.431443    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:43.433157 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:43.431517    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:27:45.431496621 +0000 UTC m=+3895.688527225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:44.406346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:44.406307    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-prpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:27:44.406519 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:44.406376    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 02 18:27:45.406261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:45.406227    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-prpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:27:45.406685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:45.406294    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 02 18:27:45.472804 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:45.472773    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:45.472804 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:45.472801    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:45.473034 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:45.472861    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:27:49.472846387 +0000 UTC m=+3899.729876985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:46.406390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:46.406349    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-prpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:27:46.406837 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:46.406428    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 02 18:27:47.406314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:47.406278    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-prpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:27:47.406509 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:47.406336    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 02 18:27:48.407135 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:48.407083    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-prpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:27:48.407585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:48.407169    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 02 18:27:49.253379 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:49.253336    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:27:49.335973 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:49.335938    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"]
Nov 02 18:27:49.406584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:49.406548    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-prpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:27:49.406775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:49.406612    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 02 18:27:49.530010 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:49.529918    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:49.530010 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:49.529953    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:49.530471 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:49.530036    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:27:57.530016254 +0000 UTC m=+3907.787046870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:49.540335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:49.540307    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:27:50.425390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:50.425360    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:27:50.429381 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:50.429349    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:27:50.547748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:50.547719    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:27:50.984221 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:50.984186    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:50.984633 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:50.984605    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:50.985420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:50.985399    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:27:54.336612 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.336571    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p" podUID="202a923e-4e0c-42c0-8680-b60162fe8e5d" containerName="payload-processing" containerID="cri-o://835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b" gracePeriod=25
Nov 02 18:27:54.607459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.607393    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:27:54.651432 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.651399    4956 generic.go:356] "Generic (PLEG): container finished" podID="202a923e-4e0c-42c0-8680-b60162fe8e5d" containerID="835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b" exitCode=0
Nov 02 18:27:54.651611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.651516    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"
Nov 02 18:27:54.651611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.651537    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p" event={"ID":"202a923e-4e0c-42c0-8680-b60162fe8e5d","Type":"ContainerDied","Data":"835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b"}
Nov 02 18:27:54.651611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.651602    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p" event={"ID":"202a923e-4e0c-42c0-8680-b60162fe8e5d","Type":"ContainerDied","Data":"7b1fd296b2a53e05a21c8f19e3ea9fa84699babb4511b2f08f9db9c301c968a0"}
Nov 02 18:27:54.651735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.651627    4956 scope.go:122] "RemoveContainer" containerID="835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b"
Nov 02 18:27:54.661451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.661432    4956 scope.go:122] "RemoveContainer" containerID="835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b"
Nov 02 18:27:54.661746 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:54.661731    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b\": container with ID starting with 835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b not found: ID does not exist" containerID="835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b"
Nov 02 18:27:54.661798 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.661755    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b"} err="failed to get container status \"835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b\": rpc error: code = NotFound desc = could not find container \"835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b\": container with ID starting with 835b201616379f71bd92f9128810f4ea7bb63065904ac0ba2ef65a0d76ca592b not found: ID does not exist"
Nov 02 18:27:54.705055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.705020    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202a923e-4e0c-42c0-8680-b60162fe8e5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a923e-4e0c-42c0-8680-b60162fe8e5d-plugins-config-volume\") pod \"202a923e-4e0c-42c0-8680-b60162fe8e5d\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") "
Nov 02 18:27:54.705264 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.705246    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202a923e-4e0c-42c0-8680-b60162fe8e5d-kube-api-access-vssbr\" (UniqueName: \"kubernetes.io/projected/202a923e-4e0c-42c0-8680-b60162fe8e5d-kube-api-access-vssbr\") pod \"202a923e-4e0c-42c0-8680-b60162fe8e5d\" (UID: \"202a923e-4e0c-42c0-8680-b60162fe8e5d\") "
Nov 02 18:27:54.705532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.705508    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202a923e-4e0c-42c0-8680-b60162fe8e5d-plugins-config-volume" pod "202a923e-4e0c-42c0-8680-b60162fe8e5d" (UID: "202a923e-4e0c-42c0-8680-b60162fe8e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:54.705851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.705827    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a923e-4e0c-42c0-8680-b60162fe8e5d-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.707618 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.707595    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202a923e-4e0c-42c0-8680-b60162fe8e5d-kube-api-access-vssbr" pod "202a923e-4e0c-42c0-8680-b60162fe8e5d" (UID: "202a923e-4e0c-42c0-8680-b60162fe8e5d"). InnerVolumeSpecName "kube-api-access-vssbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:54.807774 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:54.807734    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vssbr\" (UniqueName: \"kubernetes.io/projected/202a923e-4e0c-42c0-8680-b60162fe8e5d-kube-api-access-vssbr\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:55.073995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:55.073955    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"]
Nov 02 18:27:55.077173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:55.077149    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a72f55-658f6755d9-wdb6p"]
Nov 02 18:27:55.865868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:55.865833    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202a923e-4e0c-42c0-8680-b60162fe8e5d" path="/var/lib/kubelet/pods/202a923e-4e0c-42c0-8680-b60162fe8e5d/volumes"
Nov 02 18:27:56.495353 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:56.495312    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"]
Nov 02 18:27:56.604841 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:56.604811    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"]
Nov 02 18:27:57.545470 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:57.545436    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:57.545470 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:57.545466    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:57.546068 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:57.545537    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:28:13.54550885 +0000 UTC m=+3923.802539450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:27:59.054027 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:27:59.053984    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:27:59.252695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:27:59.252657    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:01.496097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.496000    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565" podUID="6546d7de-6ff3-4a57-a51c-d13eb7760e7d" containerName="payload-processing" containerID="cri-o://d56def61b2d21a24a58fa849a87fb517022ed172037a79b25db56a0bdbeb83b4" gracePeriod=25
Nov 02 18:28:01.605217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.605173    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8" podUID="99633e47-b96f-48cf-8cbb-5ccaba672c9b" containerName="payload-pre-processing" containerID="cri-o://ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956" gracePeriod=25
Nov 02 18:28:01.704922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.704765    4956 generic.go:356] "Generic (PLEG): container finished" podID="6546d7de-6ff3-4a57-a51c-d13eb7760e7d" containerID="d56def61b2d21a24a58fa849a87fb517022ed172037a79b25db56a0bdbeb83b4" exitCode=0
Nov 02 18:28:01.704922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.704825    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565" event={"ID":"6546d7de-6ff3-4a57-a51c-d13eb7760e7d","Type":"ContainerDied","Data":"d56def61b2d21a24a58fa849a87fb517022ed172037a79b25db56a0bdbeb83b4"}
Nov 02 18:28:01.790882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.790852    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:28:01.900637 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.900607    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-plugins-config-volume\") pod \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") "
Nov 02 18:28:01.900811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.900673    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-kube-api-access-tzvd8\" (UniqueName: \"kubernetes.io/projected/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-kube-api-access-tzvd8\") pod \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\" (UID: \"6546d7de-6ff3-4a57-a51c-d13eb7760e7d\") "
Nov 02 18:28:01.901081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.901053    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-plugins-config-volume" pod "6546d7de-6ff3-4a57-a51c-d13eb7760e7d" (UID: "6546d7de-6ff3-4a57-a51c-d13eb7760e7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:01.903080 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.903044    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-kube-api-access-tzvd8" pod "6546d7de-6ff3-4a57-a51c-d13eb7760e7d" (UID: "6546d7de-6ff3-4a57-a51c-d13eb7760e7d"). InnerVolumeSpecName "kube-api-access-tzvd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:01.906188 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:01.906166    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:28:02.002504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.002463    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99633e47-b96f-48cf-8cbb-5ccaba672c9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99633e47-b96f-48cf-8cbb-5ccaba672c9b-plugins-config-volume\") pod \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") "
Nov 02 18:28:02.002689 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.002664    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99633e47-b96f-48cf-8cbb-5ccaba672c9b-kube-api-access-mxwp9\" (UniqueName: \"kubernetes.io/projected/99633e47-b96f-48cf-8cbb-5ccaba672c9b-kube-api-access-mxwp9\") pod \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\" (UID: \"99633e47-b96f-48cf-8cbb-5ccaba672c9b\") "
Nov 02 18:28:02.002891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.002866    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99633e47-b96f-48cf-8cbb-5ccaba672c9b-plugins-config-volume" pod "99633e47-b96f-48cf-8cbb-5ccaba672c9b" (UID: "99633e47-b96f-48cf-8cbb-5ccaba672c9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:02.003030 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.003008    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.003030 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.003028    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvd8\" (UniqueName: \"kubernetes.io/projected/6546d7de-6ff3-4a57-a51c-d13eb7760e7d-kube-api-access-tzvd8\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.003235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.003038    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99633e47-b96f-48cf-8cbb-5ccaba672c9b-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.005017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.004967    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99633e47-b96f-48cf-8cbb-5ccaba672c9b-kube-api-access-mxwp9" pod "99633e47-b96f-48cf-8cbb-5ccaba672c9b" (UID: "99633e47-b96f-48cf-8cbb-5ccaba672c9b"). InnerVolumeSpecName "kube-api-access-mxwp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:02.104393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.104349    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwp9\" (UniqueName: \"kubernetes.io/projected/99633e47-b96f-48cf-8cbb-5ccaba672c9b-kube-api-access-mxwp9\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.676571 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.676535    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"]
Nov 02 18:28:02.676993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.676763    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j" podUID="4d7310ab-a339-437a-a5ed-9daecd1a882a" containerName="maas-api" containerID="cri-o://314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a" gracePeriod=30
Nov 02 18:28:02.712818 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.712778    4956 generic.go:356] "Generic (PLEG): container finished" podID="99633e47-b96f-48cf-8cbb-5ccaba672c9b" containerID="ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956" exitCode=0
Nov 02 18:28:02.712995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.712833    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"
Nov 02 18:28:02.712995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.712831    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8" event={"ID":"99633e47-b96f-48cf-8cbb-5ccaba672c9b","Type":"ContainerDied","Data":"ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956"}
Nov 02 18:28:02.712995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.712901    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8" event={"ID":"99633e47-b96f-48cf-8cbb-5ccaba672c9b","Type":"ContainerDied","Data":"88de6c18712f04cc39124e61e13e7cc382cb8b6d78ade6e276e98a0ab22af8a2"}
Nov 02 18:28:02.712995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.712932    4956 scope.go:122] "RemoveContainer" containerID="ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956"
Nov 02 18:28:02.714871 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.714851    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565" event={"ID":"6546d7de-6ff3-4a57-a51c-d13eb7760e7d","Type":"ContainerDied","Data":"72fab573274114feb64da3e4d93de5666f71ddb963a166714a3991b217632ace"}
Nov 02 18:28:02.714984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.714874    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"
Nov 02 18:28:02.791483 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.791441    4956 scope.go:122] "RemoveContainer" containerID="ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956"
Nov 02 18:28:02.792679 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:02.792649    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956\": container with ID starting with ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956 not found: ID does not exist" containerID="ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956"
Nov 02 18:28:02.792845 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.792730    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956"} err="failed to get container status \"ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956\": rpc error: code = NotFound desc = could not find container \"ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956\": container with ID starting with ab507793fca05c73161107b26142efb8b3ae9e0a6814955b1b4fb053bd9ee956 not found: ID does not exist"
Nov 02 18:28:02.792845 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.792756    4956 scope.go:122] "RemoveContainer" containerID="d56def61b2d21a24a58fa849a87fb517022ed172037a79b25db56a0bdbeb83b4"
Nov 02 18:28:02.913175 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.913141    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"]
Nov 02 18:28:02.916491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.916458    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd9ae6-9d5896b99-vtwh8"]
Nov 02 18:28:02.953726 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:02.953700    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:28:03.015087 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.015052    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-metrics-certs\") pod \"4d7310ab-a339-437a-a5ed-9daecd1a882a\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") "
Nov 02 18:28:03.015323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.015189    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7310ab-a339-437a-a5ed-9daecd1a882a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d7310ab-a339-437a-a5ed-9daecd1a882a-openshift-service-ca\") pod \"4d7310ab-a339-437a-a5ed-9daecd1a882a\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") "
Nov 02 18:28:03.015323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.015216    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls\") pod \"4d7310ab-a339-437a-a5ed-9daecd1a882a\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") "
Nov 02 18:28:03.015323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.015317    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7310ab-a339-437a-a5ed-9daecd1a882a-kube-api-access-b8p92\" (UniqueName: \"kubernetes.io/projected/4d7310ab-a339-437a-a5ed-9daecd1a882a-kube-api-access-b8p92\") pod \"4d7310ab-a339-437a-a5ed-9daecd1a882a\" (UID: \"4d7310ab-a339-437a-a5ed-9daecd1a882a\") "
Nov 02 18:28:03.015599 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.015575    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7310ab-a339-437a-a5ed-9daecd1a882a-openshift-service-ca" pod "4d7310ab-a339-437a-a5ed-9daecd1a882a" (UID: "4d7310ab-a339-437a-a5ed-9daecd1a882a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:03.015766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.015750    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d7310ab-a339-437a-a5ed-9daecd1a882a-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:03.017505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.017483    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls" pod "4d7310ab-a339-437a-a5ed-9daecd1a882a" (UID: "4d7310ab-a339-437a-a5ed-9daecd1a882a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:03.017595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.017582    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-metrics-certs" pod "4d7310ab-a339-437a-a5ed-9daecd1a882a" (UID: "4d7310ab-a339-437a-a5ed-9daecd1a882a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:03.017635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.017585    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7310ab-a339-437a-a5ed-9daecd1a882a-kube-api-access-b8p92" pod "4d7310ab-a339-437a-a5ed-9daecd1a882a" (UID: "4d7310ab-a339-437a-a5ed-9daecd1a882a"). InnerVolumeSpecName "kube-api-access-b8p92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:03.022886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.022857    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"]
Nov 02 18:28:03.025960 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.025934    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd9ae6-d9c576745-bx565"]
Nov 02 18:28:03.116909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.116810    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8p92\" (UniqueName: \"kubernetes.io/projected/4d7310ab-a339-437a-a5ed-9daecd1a882a-kube-api-access-b8p92\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:03.116909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.116843    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:03.116909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.116852    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d7310ab-a339-437a-a5ed-9daecd1a882a-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:28:03.723949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.723921    4956 generic.go:356] "Generic (PLEG): container finished" podID="4d7310ab-a339-437a-a5ed-9daecd1a882a" containerID="314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a" exitCode=0
Nov 02 18:28:03.724442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.724045    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"
Nov 02 18:28:03.724442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.724047    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j" event={"ID":"4d7310ab-a339-437a-a5ed-9daecd1a882a","Type":"ContainerDied","Data":"314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a"}
Nov 02 18:28:03.724442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.724181    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j" event={"ID":"4d7310ab-a339-437a-a5ed-9daecd1a882a","Type":"ContainerDied","Data":"41458720d31addb1351e0b95ef3c56b790b76e48b120423359b73bdc5e2756ab"}
Nov 02 18:28:03.724442 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.724217    4956 scope.go:122] "RemoveContainer" containerID="314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a"
Nov 02 18:28:03.743995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.743967    4956 scope.go:122] "RemoveContainer" containerID="314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a"
Nov 02 18:28:03.744362 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:03.744340    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a\": container with ID starting with 314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a not found: ID does not exist" containerID="314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a"
Nov 02 18:28:03.744427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.744373    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a"} err="failed to get container status \"314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a\": rpc error: code = NotFound desc = could not find container \"314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a\": container with ID starting with 314f27b1518a93d28d73162762d053a875abdcb0c01b8dcc3a3a6150a66ff72a not found: ID does not exist"
Nov 02 18:28:03.865844 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.865809    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6546d7de-6ff3-4a57-a51c-d13eb7760e7d" path="/var/lib/kubelet/pods/6546d7de-6ff3-4a57-a51c-d13eb7760e7d/volumes"
Nov 02 18:28:03.866309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.866295    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99633e47-b96f-48cf-8cbb-5ccaba672c9b" path="/var/lib/kubelet/pods/99633e47-b96f-48cf-8cbb-5ccaba672c9b/volumes"
Nov 02 18:28:03.892316 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.892277    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"]
Nov 02 18:28:03.931167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:03.931132    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd9ae6-f88d898f4-96v6j"]
Nov 02 18:28:05.867654 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:05.867621    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7310ab-a339-437a-a5ed-9daecd1a882a" path="/var/lib/kubelet/pods/4d7310ab-a339-437a-a5ed-9daecd1a882a/volumes"
Nov 02 18:28:09.252822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:09.252783    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:13.551372 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:13.551333    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:28:13.551372 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:13.551361    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:28:13.551914 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:13.551423    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:28:45.551405065 +0000 UTC m=+3955.808435662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:28:18.876648 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.876604    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"]
Nov 02 18:28:18.877482 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877460    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4d7310ab-a339-437a-a5ed-9daecd1a882a" containerName="maas-api"
Nov 02 18:28:18.877604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877483    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7310ab-a339-437a-a5ed-9daecd1a882a" containerName="maas-api"
Nov 02 18:28:18.877604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877507    4956 cpu_manager.go:414] "containerMap: removing container" podUID="99633e47-b96f-48cf-8cbb-5ccaba672c9b" containerName="payload-pre-processing"
Nov 02 18:28:18.877604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877517    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99633e47-b96f-48cf-8cbb-5ccaba672c9b" containerName="payload-pre-processing"
Nov 02 18:28:18.877604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877541    4956 cpu_manager.go:414] "containerMap: removing container" podUID="202a923e-4e0c-42c0-8680-b60162fe8e5d" containerName="payload-processing"
Nov 02 18:28:18.877604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877548    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202a923e-4e0c-42c0-8680-b60162fe8e5d" containerName="payload-processing"
Nov 02 18:28:18.877604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877575    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6546d7de-6ff3-4a57-a51c-d13eb7760e7d" containerName="payload-processing"
Nov 02 18:28:18.877604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877583    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6546d7de-6ff3-4a57-a51c-d13eb7760e7d" containerName="payload-processing"
Nov 02 18:28:18.877950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877761    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7310ab-a339-437a-a5ed-9daecd1a882a" containerName="maas-api"
Nov 02 18:28:18.877950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877777    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="99633e47-b96f-48cf-8cbb-5ccaba672c9b" containerName="payload-pre-processing"
Nov 02 18:28:18.877950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877790    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="202a923e-4e0c-42c0-8680-b60162fe8e5d" containerName="payload-processing"
Nov 02 18:28:18.877950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.877805    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6546d7de-6ff3-4a57-a51c-d13eb7760e7d" containerName="payload-processing"
Nov 02 18:28:18.881591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.881569    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:18.937270 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.937227    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"]
Nov 02 18:28:18.982732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.982698    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-740c43\""
Nov 02 18:28:18.982732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:18.982698    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-740c43-dockercfg-w8qgx\""
Nov 02 18:28:19.019535 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.019501    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60844996-53ca-4873-a67e-14d51a48f8b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.019821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.019797    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2kmb\" (UniqueName: \"kubernetes.io/projected/60844996-53ca-4873-a67e-14d51a48f8b6-kube-api-access-n2kmb\") pod \"payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.037657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.037624    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"]
Nov 02 18:28:19.043148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.043107    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.081240 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.081208    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"]
Nov 02 18:28:19.121494 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.121450    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcf7j\" (UniqueName: \"kubernetes.io/projected/546e4fdc-ee01-438f-9c4e-3bc4a611536a-kube-api-access-kcf7j\") pod \"payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.121700 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.121585    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2kmb\" (UniqueName: \"kubernetes.io/projected/60844996-53ca-4873-a67e-14d51a48f8b6-kube-api-access-n2kmb\") pod \"payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.121763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.121734    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60844996-53ca-4873-a67e-14d51a48f8b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.121831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.121801    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546e4fdc-ee01-438f-9c4e-3bc4a611536a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.122764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.122736    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60844996-53ca-4873-a67e-14d51a48f8b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.179166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.179074    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2kmb\" (UniqueName: \"kubernetes.io/projected/60844996-53ca-4873-a67e-14d51a48f8b6-kube-api-access-n2kmb\") pod \"payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.196008 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.195979    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.223477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.223447    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcf7j\" (UniqueName: \"kubernetes.io/projected/546e4fdc-ee01-438f-9c4e-3bc4a611536a-kube-api-access-kcf7j\") pod \"payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.223678 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.223657    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546e4fdc-ee01-438f-9c4e-3bc4a611536a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.224392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.224365    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546e4fdc-ee01-438f-9c4e-3bc4a611536a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.253592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.253559    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:19.283159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.280511    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcf7j\" (UniqueName: \"kubernetes.io/projected/546e4fdc-ee01-438f-9c4e-3bc4a611536a-kube-api-access-kcf7j\") pod \"payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.358175 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.358136    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.585660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.585627    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"]
Nov 02 18:28:19.586565 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:28:19.586538    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60844996_53ca_4873_a67e_14d51a48f8b6.slice/crio-d0590883c752df685cde683ceff79f8e6bbe6a00850653b61389531133b232f3 WatchSource:0}: Error finding container d0590883c752df685cde683ceff79f8e6bbe6a00850653b61389531133b232f3: Status 404 returned error can't find the container with id d0590883c752df685cde683ceff79f8e6bbe6a00850653b61389531133b232f3
Nov 02 18:28:19.735631 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:28:19.735584    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546e4fdc_ee01_438f_9c4e_3bc4a611536a.slice/crio-a02e72c195fe3265c0b1f190a27b9ab478db2baf369fc383ba998a4cb522484d WatchSource:0}: Error finding container a02e72c195fe3265c0b1f190a27b9ab478db2baf369fc383ba998a4cb522484d: Status 404 returned error can't find the container with id a02e72c195fe3265c0b1f190a27b9ab478db2baf369fc383ba998a4cb522484d
Nov 02 18:28:19.737100 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.737077    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"]
Nov 02 18:28:19.842263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.842169    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl" event={"ID":"60844996-53ca-4873-a67e-14d51a48f8b6","Type":"ContainerStarted","Data":"f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd"}
Nov 02 18:28:19.842263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.842237    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl" event={"ID":"60844996-53ca-4873-a67e-14d51a48f8b6","Type":"ContainerStarted","Data":"d0590883c752df685cde683ceff79f8e6bbe6a00850653b61389531133b232f3"}
Nov 02 18:28:19.842459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.842393    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:19.843937 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.843911    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb" event={"ID":"546e4fdc-ee01-438f-9c4e-3bc4a611536a","Type":"ContainerStarted","Data":"4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5"}
Nov 02 18:28:19.844052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.843960    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:19.844052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.843978    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb" event={"ID":"546e4fdc-ee01-438f-9c4e-3bc4a611536a","Type":"ContainerStarted","Data":"a02e72c195fe3265c0b1f190a27b9ab478db2baf369fc383ba998a4cb522484d"}
Nov 02 18:28:19.956309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:19.956258    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl" podStartSLOduration=1.9562384609999999 podStartE2EDuration="1.956238461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:19.955283819 +0000 UTC m=+3930.212314439" watchObservedRunningTime="2026-11-02 18:28:19.956238461 +0000 UTC m=+3930.213269082"
Nov 02 18:28:20.073851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:20.073784    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb" podStartSLOduration=2.073767078 podStartE2EDuration="2.073767078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:20.068672238 +0000 UTC m=+3930.325702850" watchObservedRunningTime="2026-11-02 18:28:20.073767078 +0000 UTC m=+3930.330797698"
Nov 02 18:28:26.858732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:26.858687    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:29.053034 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:29.052997    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:28:30.855392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:30.855362    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:28:30.855837 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:30.855523    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:28:36.858833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:36.858796    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:45.582721 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:45.582685    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:28:45.582721 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:45.582712    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:28:45.583253 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:45.582770    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:29:49.582755966 +0000 UTC m=+4019.839786578 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:28:46.858717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:46.858679    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:56.858872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:56.858836    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:58.577663 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.577625    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"]
Nov 02 18:28:58.581954 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.581934    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.655510 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.655475    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-metrics-certs\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.655710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.655583    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.655710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.655613    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2w29\" (UniqueName: \"kubernetes.io/projected/780d4861-313a-4b69-bf27-ce324a57639a-kube-api-access-j2w29\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.655710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.655672    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780d4861-313a-4b69-bf27-ce324a57639a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.684872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.684835    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-546881-serving-cert\""
Nov 02 18:28:58.688065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.688042    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"]
Nov 02 18:28:58.757584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.757547    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-metrics-certs\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.757763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.757605    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.757763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.757633    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2w29\" (UniqueName: \"kubernetes.io/projected/780d4861-313a-4b69-bf27-ce324a57639a-kube-api-access-j2w29\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.757763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.757656    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780d4861-313a-4b69-bf27-ce324a57639a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.757890 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:58.757775    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-serving-cert: secret "maas-api-e2e-inf-a-546881-serving-cert" not found
Nov 02 18:28:58.757890 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:58.757852    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls podName:780d4861-313a-4b69-bf27-ce324a57639a nodeName:}" failed. No retries permitted until 2026-11-02 18:28:59.25783394 +0000 UTC m=+3969.514864544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls") pod "maas-api-e2e-inf-a-546881-58b7946789-6zmdw" (UID: "780d4861-313a-4b69-bf27-ce324a57639a") : secret "maas-api-e2e-inf-a-546881-serving-cert" not found
Nov 02 18:28:58.758457 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.758434    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780d4861-313a-4b69-bf27-ce324a57639a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.760225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.760205    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-metrics-certs\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:58.815229 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:58.815195    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2w29\" (UniqueName: \"kubernetes.io/projected/780d4861-313a-4b69-bf27-ce324a57639a-kube-api-access-j2w29\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:59.056372 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:28:59.056336    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:28:59.263867 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:59.263822    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:59.266462 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:59.266440    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls\") pod \"maas-api-e2e-inf-a-546881-58b7946789-6zmdw\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:59.495872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:59.495837    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:28:59.880709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:59.880679    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"]
Nov 02 18:28:59.881962 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:28:59.881934    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780d4861_313a_4b69_bf27_ce324a57639a.slice/crio-f8110b866e5757ce138fcc54c5e7a3cb9c24de537f48d5673ce74027977a98e2 WatchSource:0}: Error finding container f8110b866e5757ce138fcc54c5e7a3cb9c24de537f48d5673ce74027977a98e2: Status 404 returned error can't find the container with id f8110b866e5757ce138fcc54c5e7a3cb9c24de537f48d5673ce74027977a98e2
Nov 02 18:28:59.883325 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:28:59.883302    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:29:00.154446 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:00.154353    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw" event={"ID":"780d4861-313a-4b69-bf27-ce324a57639a","Type":"ContainerStarted","Data":"f8110b866e5757ce138fcc54c5e7a3cb9c24de537f48d5673ce74027977a98e2"}
Nov 02 18:29:01.163971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:01.163888    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw" event={"ID":"780d4861-313a-4b69-bf27-ce324a57639a","Type":"ContainerStarted","Data":"9ebe694ae7ea1bc599a82b56780d62db0d559a42fafccc06915b5b9c4c9baed2"}
Nov 02 18:29:01.164370 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:01.163999    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:29:01.282658 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:01.282602    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw" podStartSLOduration=2.442407619 podStartE2EDuration="3.282589528s" totalImagesPullingTime="840.181909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:01.278657453 +0000 UTC m=+3971.535688073" watchObservedRunningTime="2026-11-02 18:29:01.282589528 +0000 UTC m=+3971.539620148"
Nov 02 18:29:06.858793 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:06.858757    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:07.176595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:07.176512    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:29:16.858967 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:16.858931    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:26.858609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:26.858567    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:29.053669 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:29:29.053634    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:29:36.858826 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:36.858787    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:42.859141 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:42.859091    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:49.625130 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:29:49.625090    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:29:49.625543 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:29:49.625149    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:29:49.625543 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:29:49.625235    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:31:51.625220962 +0000 UTC m=+4141.882251580 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:29:52.859172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:29:52.859089    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:59.053554 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:29:59.053516    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:30:00.191833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.191794    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"]
Nov 02 18:30:00.196464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.196436    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.199563 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.199532    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"]
Nov 02 18:30:00.206907 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.206883    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.222320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.222282    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"]
Nov 02 18:30:00.226680 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.226656    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.241833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.241810    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"]
Nov 02 18:30:00.246095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.246074    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.246857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.246835    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"]
Nov 02 18:30:00.251317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.251297    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.264002 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.263969    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8tl\" (UniqueName: \"kubernetes.io/projected/81694129-39c4-412d-bcaf-ab281c89be8b-kube-api-access-ns8tl\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.264219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.264015    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzj2t\" (UniqueName: \"kubernetes.io/projected/865e7494-ecdc-4041-a209-35b47628e625-kube-api-access-dzj2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.264219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.264066    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81694129-39c4-412d-bcaf-ab281c89be8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.264370 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.264230    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865e7494-ecdc-4041-a209-35b47628e625-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.276460 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.276418    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"]
Nov 02 18:30:00.283327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.283295    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"]
Nov 02 18:30:00.323899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.323864    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"]
Nov 02 18:30:00.327634 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.327605    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"]
Nov 02 18:30:00.330277 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.330246    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:30:00.342415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.342375    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"]
Nov 02 18:30:00.365719 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.365681    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrr2\" (UniqueName: \"kubernetes.io/projected/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-kube-api-access-jhrr2\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.365914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.365744    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.365914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.365837    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhgc\" (UniqueName: \"kubernetes.io/projected/1ee52229-2f22-4de4-973b-df0f5d917cb6-kube-api-access-tkhgc\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.365914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.365878    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee52229-2f22-4de4-973b-df0f5d917cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.366038 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.365978    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8tl\" (UniqueName: \"kubernetes.io/projected/81694129-39c4-412d-bcaf-ab281c89be8b-kube-api-access-ns8tl\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.366038 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.366013    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzj2t\" (UniqueName: \"kubernetes.io/projected/865e7494-ecdc-4041-a209-35b47628e625-kube-api-access-dzj2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.366170 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.366034    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6226cfb6-2712-4c01-9d36-baa4e69a749f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.366170 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.366074    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81694129-39c4-412d-bcaf-ab281c89be8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.366170 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.366153    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865e7494-ecdc-4041-a209-35b47628e625-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.366324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.366223    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pzp\" (UniqueName: \"kubernetes.io/projected/6226cfb6-2712-4c01-9d36-baa4e69a749f-kube-api-access-55pzp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.366959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.366936    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865e7494-ecdc-4041-a209-35b47628e625-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.367108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.367003    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81694129-39c4-412d-bcaf-ab281c89be8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.425728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.425688    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8tl\" (UniqueName: \"kubernetes.io/projected/81694129-39c4-412d-bcaf-ab281c89be8b-kube-api-access-ns8tl\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.425728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.425728    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzj2t\" (UniqueName: \"kubernetes.io/projected/865e7494-ecdc-4041-a209-35b47628e625-kube-api-access-dzj2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.468050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.468006    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6226cfb6-2712-4c01-9d36-baa4e69a749f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.468291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.468101    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pzp\" (UniqueName: \"kubernetes.io/projected/6226cfb6-2712-4c01-9d36-baa4e69a749f-kube-api-access-55pzp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.468367 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.468287    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrr2\" (UniqueName: \"kubernetes.io/projected/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-kube-api-access-jhrr2\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.468367 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.468338    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.468476 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.468394    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhgc\" (UniqueName: \"kubernetes.io/projected/1ee52229-2f22-4de4-973b-df0f5d917cb6-kube-api-access-tkhgc\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.468476 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.468443    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee52229-2f22-4de4-973b-df0f5d917cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.468869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.468844    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6226cfb6-2712-4c01-9d36-baa4e69a749f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.469031 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.469010    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.469095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.469048    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee52229-2f22-4de4-973b-df0f5d917cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.511525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.511493    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:00.523301 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.523273    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:00.528481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.528447    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhgc\" (UniqueName: \"kubernetes.io/projected/1ee52229-2f22-4de4-973b-df0f5d917cb6-kube-api-access-tkhgc\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.528923 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.528899    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pzp\" (UniqueName: \"kubernetes.io/projected/6226cfb6-2712-4c01-9d36-baa4e69a749f-kube-api-access-55pzp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.529309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.529281    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrr2\" (UniqueName: \"kubernetes.io/projected/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-kube-api-access-jhrr2\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.543325 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.543299    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:00.569733 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.569694    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:00.590902 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.590865    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:00.907305 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.907276    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"]
Nov 02 18:30:00.911156 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:30:00.909780    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865e7494_ecdc_4041_a209_35b47628e625.slice/crio-8eda2f85f2f860ab96906277b5a928b91b81f0d7aef4a4643cdc736bdee06329 WatchSource:0}: Error finding container 8eda2f85f2f860ab96906277b5a928b91b81f0d7aef4a4643cdc736bdee06329: Status 404 returned error can't find the container with id 8eda2f85f2f860ab96906277b5a928b91b81f0d7aef4a4643cdc736bdee06329
Nov 02 18:30:00.955247 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.955217    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"]
Nov 02 18:30:00.957099 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:30:00.957034    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81694129_39c4_412d_bcaf_ab281c89be8b.slice/crio-c0d165c07261603b465782cb9e654c8a227e05073a17b797bc64b149a6d66ed1 WatchSource:0}: Error finding container c0d165c07261603b465782cb9e654c8a227e05073a17b797bc64b149a6d66ed1: Status 404 returned error can't find the container with id c0d165c07261603b465782cb9e654c8a227e05073a17b797bc64b149a6d66ed1
Nov 02 18:30:00.959661 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:00.959637    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"]
Nov 02 18:30:00.959771 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:30:00.959742    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6226cfb6_2712_4c01_9d36_baa4e69a749f.slice/crio-390efe42327a2ffb71510826ca0dbc197aae445ed8034fdabf85708ee66f93a3 WatchSource:0}: Error finding container 390efe42327a2ffb71510826ca0dbc197aae445ed8034fdabf85708ee66f93a3: Status 404 returned error can't find the container with id 390efe42327a2ffb71510826ca0dbc197aae445ed8034fdabf85708ee66f93a3
Nov 02 18:30:01.007107 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.007083    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"]
Nov 02 18:30:01.007902 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:30:01.007877    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee52229_2f22_4de4_973b_df0f5d917cb6.slice/crio-59f0f422628eb151a00103ba8ea9a0b66d543c0e45475a8da08ae7897c60ea1d WatchSource:0}: Error finding container 59f0f422628eb151a00103ba8ea9a0b66d543c0e45475a8da08ae7897c60ea1d: Status 404 returned error can't find the container with id 59f0f422628eb151a00103ba8ea9a0b66d543c0e45475a8da08ae7897c60ea1d
Nov 02 18:30:01.009620 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:30:01.009591    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd610f966_10aa_4f0c_a3cc_e16bce11ba4e.slice/crio-376ff6a8412720a4a39f6284c6c8697a51f5792aff92666420c22931779e91d6 WatchSource:0}: Error finding container 376ff6a8412720a4a39f6284c6c8697a51f5792aff92666420c22931779e91d6: Status 404 returned error can't find the container with id 376ff6a8412720a4a39f6284c6c8697a51f5792aff92666420c22931779e91d6
Nov 02 18:30:01.011030 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.010976    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"]
Nov 02 18:30:01.564582 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.564469    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerStarted","Data":"d0d814ceea4c2524fb7b138a4d46681a5d905da1ab97aa6063bc90ec422dc65e"}
Nov 02 18:30:01.564582 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.564544    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerStarted","Data":"c0d165c07261603b465782cb9e654c8a227e05073a17b797bc64b149a6d66ed1"}
Nov 02 18:30:01.566159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.566132    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" event={"ID":"1ee52229-2f22-4de4-973b-df0f5d917cb6","Type":"ContainerStarted","Data":"59f0f422628eb151a00103ba8ea9a0b66d543c0e45475a8da08ae7897c60ea1d"}
Nov 02 18:30:01.567823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.567799    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" event={"ID":"d610f966-10aa-4f0c-a3cc-e16bce11ba4e","Type":"ContainerStarted","Data":"441d4d14f4b11149aafc6084e0845ec90cb003fb051201cdf03b9b005a464e04"}
Nov 02 18:30:01.567960 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.567848    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" event={"ID":"d610f966-10aa-4f0c-a3cc-e16bce11ba4e","Type":"ContainerStarted","Data":"376ff6a8412720a4a39f6284c6c8697a51f5792aff92666420c22931779e91d6"}
Nov 02 18:30:01.569440 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.569407    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerStarted","Data":"7f13a4fcdd38e95ede6b0deb0215b01131979e4df8c6611d841e5fa5533b9bb3"}
Nov 02 18:30:01.569563 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.569460    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerStarted","Data":"390efe42327a2ffb71510826ca0dbc197aae445ed8034fdabf85708ee66f93a3"}
Nov 02 18:30:01.571028 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.571009    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerStarted","Data":"e1961c88ec2550e758058fedd606e84e3e9d4f5792440743b256fee570b0deae"}
Nov 02 18:30:01.571160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.571043    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerStarted","Data":"8eda2f85f2f860ab96906277b5a928b91b81f0d7aef4a4643cdc736bdee06329"}
Nov 02 18:30:01.680161 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.680087    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" podStartSLOduration=1.680067587 podStartE2EDuration="1.680067587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.67932505 +0000 UTC m=+4031.936355670" watchObservedRunningTime="2026-11-02 18:30:01.680067587 +0000 UTC m=+4031.937098211"
Nov 02 18:30:01.794517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.793564    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" podStartSLOduration=1.793548782 podStartE2EDuration="1.793548782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.792316138 +0000 UTC m=+4032.049346764" watchObservedRunningTime="2026-11-02 18:30:01.793548782 +0000 UTC m=+4032.050579402"
Nov 02 18:30:01.917509 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:01.917403    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" podStartSLOduration=1.917387482 podStartE2EDuration="1.917387482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.917026012 +0000 UTC m=+4032.174056646" watchObservedRunningTime="2026-11-02 18:30:01.917387482 +0000 UTC m=+4032.174418102"
Nov 02 18:30:02.033694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.033609    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" podStartSLOduration=2.033589622 podStartE2EDuration="2.033589622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:02.032799275 +0000 UTC m=+4032.289829895" watchObservedRunningTime="2026-11-02 18:30:02.033589622 +0000 UTC m=+4032.290620243"
Nov 02 18:30:02.579093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.578999    4956 generic.go:356] "Generic (PLEG): container finished" podID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerID="39043d50e7ab2305eb7e260f73735cc6e5003976b25c4c99a5559b807135cdff" exitCode=60
Nov 02 18:30:02.579093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.579078    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" event={"ID":"1ee52229-2f22-4de4-973b-df0f5d917cb6","Type":"ContainerDied","Data":"39043d50e7ab2305eb7e260f73735cc6e5003976b25c4c99a5559b807135cdff"}
Nov 02 18:30:02.579591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.579459    4956 scope.go:122] "RemoveContainer" containerID="39043d50e7ab2305eb7e260f73735cc6e5003976b25c4c99a5559b807135cdff"
Nov 02 18:30:02.580904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.580881    4956 generic.go:356] "Generic (PLEG): container finished" podID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerID="441d4d14f4b11149aafc6084e0845ec90cb003fb051201cdf03b9b005a464e04" exitCode=60
Nov 02 18:30:02.580998 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.580972    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" event={"ID":"d610f966-10aa-4f0c-a3cc-e16bce11ba4e","Type":"ContainerDied","Data":"441d4d14f4b11149aafc6084e0845ec90cb003fb051201cdf03b9b005a464e04"}
Nov 02 18:30:02.581256 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.581220    4956 scope.go:122] "RemoveContainer" containerID="441d4d14f4b11149aafc6084e0845ec90cb003fb051201cdf03b9b005a464e04"
Nov 02 18:30:02.582734 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.582626    4956 generic.go:356] "Generic (PLEG): container finished" podID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerID="7f13a4fcdd38e95ede6b0deb0215b01131979e4df8c6611d841e5fa5533b9bb3" exitCode=60
Nov 02 18:30:02.582734 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.582684    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerDied","Data":"7f13a4fcdd38e95ede6b0deb0215b01131979e4df8c6611d841e5fa5533b9bb3"}
Nov 02 18:30:02.582953 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.582934    4956 scope.go:122] "RemoveContainer" containerID="7f13a4fcdd38e95ede6b0deb0215b01131979e4df8c6611d841e5fa5533b9bb3"
Nov 02 18:30:02.584573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.584456    4956 generic.go:356] "Generic (PLEG): container finished" podID="865e7494-ecdc-4041-a209-35b47628e625" containerID="e1961c88ec2550e758058fedd606e84e3e9d4f5792440743b256fee570b0deae" exitCode=60
Nov 02 18:30:02.584573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.584525    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerDied","Data":"e1961c88ec2550e758058fedd606e84e3e9d4f5792440743b256fee570b0deae"}
Nov 02 18:30:02.584689 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.584672    4956 scope.go:122] "RemoveContainer" containerID="e1961c88ec2550e758058fedd606e84e3e9d4f5792440743b256fee570b0deae"
Nov 02 18:30:02.586307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.586291    4956 generic.go:356] "Generic (PLEG): container finished" podID="81694129-39c4-412d-bcaf-ab281c89be8b" containerID="d0d814ceea4c2524fb7b138a4d46681a5d905da1ab97aa6063bc90ec422dc65e" exitCode=60
Nov 02 18:30:02.586396 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.586354    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerDied","Data":"d0d814ceea4c2524fb7b138a4d46681a5d905da1ab97aa6063bc90ec422dc65e"}
Nov 02 18:30:02.586524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.586509    4956 scope.go:122] "RemoveContainer" containerID="d0d814ceea4c2524fb7b138a4d46681a5d905da1ab97aa6063bc90ec422dc65e"
Nov 02 18:30:02.859643 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:02.859609    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:03.594892 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.594854    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerStarted","Data":"7f88e0c5ab1d1fa128cfe64c393e4eef7302c07ed6e8a3cbdb6df77efdc2ac32"}
Nov 02 18:30:03.596985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.596952    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerStarted","Data":"b6b5089d470698561960b0b88a8855e21ad8708a43de6496e5f852fc9dc4c027"}
Nov 02 18:30:03.598950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.598920    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerStarted","Data":"d199e233ab634507148c105e24fb9259576b8a29c7c6aee3ed2e10ef72310f18"}
Nov 02 18:30:03.600765 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.600744    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" event={"ID":"1ee52229-2f22-4de4-973b-df0f5d917cb6","Type":"ContainerStarted","Data":"5779089d23fbcb165e6b892d89f1ea87fbd28321f10b4b5d356e1a1cf2da8756"}
Nov 02 18:30:03.602598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.602577    4956 generic.go:356] "Generic (PLEG): container finished" podID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerID="0897a0c579f518179836c880a50fa1361f3333ee276ca46e759282943171f583" exitCode=60
Nov 02 18:30:03.602686 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.602611    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" event={"ID":"d610f966-10aa-4f0c-a3cc-e16bce11ba4e","Type":"ContainerDied","Data":"0897a0c579f518179836c880a50fa1361f3333ee276ca46e759282943171f583"}
Nov 02 18:30:03.602686 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.602646    4956 scope.go:122] "RemoveContainer" containerID="441d4d14f4b11149aafc6084e0845ec90cb003fb051201cdf03b9b005a464e04"
Nov 02 18:30:03.602873 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.602850    4956 scope.go:122] "RemoveContainer" containerID="0897a0c579f518179836c880a50fa1361f3333ee276ca46e759282943171f583"
Nov 02 18:30:03.603136 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:03.603098    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp_odh-ai-gateway-infra(d610f966-10aa-4f0c-a3cc-e16bce11ba4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e"
Nov 02 18:30:03.944699 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:03.944653    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" podStartSLOduration=3.944637712 podStartE2EDuration="3.944637712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.941062529 +0000 UTC m=+4034.198093148" watchObservedRunningTime="2026-11-02 18:30:03.944637712 +0000 UTC m=+4034.201668332"
Nov 02 18:30:04.610872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.610825    4956 generic.go:356] "Generic (PLEG): container finished" podID="81694129-39c4-412d-bcaf-ab281c89be8b" containerID="d199e233ab634507148c105e24fb9259576b8a29c7c6aee3ed2e10ef72310f18" exitCode=60
Nov 02 18:30:04.611495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.610884    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerDied","Data":"d199e233ab634507148c105e24fb9259576b8a29c7c6aee3ed2e10ef72310f18"}
Nov 02 18:30:04.611495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.610989    4956 scope.go:122] "RemoveContainer" containerID="d0d814ceea4c2524fb7b138a4d46681a5d905da1ab97aa6063bc90ec422dc65e"
Nov 02 18:30:04.611495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.611332    4956 scope.go:122] "RemoveContainer" containerID="d199e233ab634507148c105e24fb9259576b8a29c7c6aee3ed2e10ef72310f18"
Nov 02 18:30:04.611689 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:04.611669    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh_odh-ai-gateway-infra(81694129-39c4-412d-bcaf-ab281c89be8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" podUID="81694129-39c4-412d-bcaf-ab281c89be8b"
Nov 02 18:30:04.613221 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.613201    4956 generic.go:356] "Generic (PLEG): container finished" podID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerID="5779089d23fbcb165e6b892d89f1ea87fbd28321f10b4b5d356e1a1cf2da8756" exitCode=60
Nov 02 18:30:04.613314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.613283    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" event={"ID":"1ee52229-2f22-4de4-973b-df0f5d917cb6","Type":"ContainerDied","Data":"5779089d23fbcb165e6b892d89f1ea87fbd28321f10b4b5d356e1a1cf2da8756"}
Nov 02 18:30:04.613544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.613526    4956 scope.go:122] "RemoveContainer" containerID="5779089d23fbcb165e6b892d89f1ea87fbd28321f10b4b5d356e1a1cf2da8756"
Nov 02 18:30:04.613846 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:04.613828    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq_odh-ai-gateway-infra(1ee52229-2f22-4de4-973b-df0f5d917cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6"
Nov 02 18:30:04.615022 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.614995    4956 scope.go:122] "RemoveContainer" containerID="0897a0c579f518179836c880a50fa1361f3333ee276ca46e759282943171f583"
Nov 02 18:30:04.615264 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:04.615245    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp_odh-ai-gateway-infra(d610f966-10aa-4f0c-a3cc-e16bce11ba4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e"
Nov 02 18:30:04.616762 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.616742    4956 generic.go:356] "Generic (PLEG): container finished" podID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerID="7f88e0c5ab1d1fa128cfe64c393e4eef7302c07ed6e8a3cbdb6df77efdc2ac32" exitCode=60
Nov 02 18:30:04.616857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.616827    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerDied","Data":"7f88e0c5ab1d1fa128cfe64c393e4eef7302c07ed6e8a3cbdb6df77efdc2ac32"}
Nov 02 18:30:04.616996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.616981    4956 scope.go:122] "RemoveContainer" containerID="7f88e0c5ab1d1fa128cfe64c393e4eef7302c07ed6e8a3cbdb6df77efdc2ac32"
Nov 02 18:30:04.617241 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:04.617219    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm_odh-ai-gateway-infra(6226cfb6-2712-4c01-9d36-baa4e69a749f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f"
Nov 02 18:30:04.618702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.618681    4956 generic.go:356] "Generic (PLEG): container finished" podID="865e7494-ecdc-4041-a209-35b47628e625" containerID="b6b5089d470698561960b0b88a8855e21ad8708a43de6496e5f852fc9dc4c027" exitCode=60
Nov 02 18:30:04.618799 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.618718    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerDied","Data":"b6b5089d470698561960b0b88a8855e21ad8708a43de6496e5f852fc9dc4c027"}
Nov 02 18:30:04.618968 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.618953    4956 scope.go:122] "RemoveContainer" containerID="b6b5089d470698561960b0b88a8855e21ad8708a43de6496e5f852fc9dc4c027"
Nov 02 18:30:04.619186 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:04.619168    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k_odh-ai-gateway-infra(865e7494-ecdc-4041-a209-35b47628e625)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" podUID="865e7494-ecdc-4041-a209-35b47628e625"
Nov 02 18:30:04.624562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.624545    4956 scope.go:122] "RemoveContainer" containerID="39043d50e7ab2305eb7e260f73735cc6e5003976b25c4c99a5559b807135cdff"
Nov 02 18:30:04.637911 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.637888    4956 scope.go:122] "RemoveContainer" containerID="7f13a4fcdd38e95ede6b0deb0215b01131979e4df8c6611d841e5fa5533b9bb3"
Nov 02 18:30:04.651459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:04.651436    4956 scope.go:122] "RemoveContainer" containerID="e1961c88ec2550e758058fedd606e84e3e9d4f5792440743b256fee570b0deae"
Nov 02 18:30:05.627394 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:05.627357    4956 scope.go:122] "RemoveContainer" containerID="d199e233ab634507148c105e24fb9259576b8a29c7c6aee3ed2e10ef72310f18"
Nov 02 18:30:05.627823 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:05.627706    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh_odh-ai-gateway-infra(81694129-39c4-412d-bcaf-ab281c89be8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" podUID="81694129-39c4-412d-bcaf-ab281c89be8b"
Nov 02 18:30:05.628987 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:05.628967    4956 scope.go:122] "RemoveContainer" containerID="5779089d23fbcb165e6b892d89f1ea87fbd28321f10b4b5d356e1a1cf2da8756"
Nov 02 18:30:05.629337 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:05.629315    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq_odh-ai-gateway-infra(1ee52229-2f22-4de4-973b-df0f5d917cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6"
Nov 02 18:30:05.630628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:05.630610    4956 scope.go:122] "RemoveContainer" containerID="7f88e0c5ab1d1fa128cfe64c393e4eef7302c07ed6e8a3cbdb6df77efdc2ac32"
Nov 02 18:30:05.630846 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:05.630827    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm_odh-ai-gateway-infra(6226cfb6-2712-4c01-9d36-baa4e69a749f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f"
Nov 02 18:30:05.632023 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:05.632009    4956 scope.go:122] "RemoveContainer" containerID="b6b5089d470698561960b0b88a8855e21ad8708a43de6496e5f852fc9dc4c027"
Nov 02 18:30:05.632245 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:05.632227    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k_odh-ai-gateway-infra(865e7494-ecdc-4041-a209-35b47628e625)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" podUID="865e7494-ecdc-4041-a209-35b47628e625"
Nov 02 18:30:12.859352 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:12.859311    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:14.857882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:14.857839    4956 scope.go:122] "RemoveContainer" containerID="7f88e0c5ab1d1fa128cfe64c393e4eef7302c07ed6e8a3cbdb6df77efdc2ac32"
Nov 02 18:30:14.858382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:14.857970    4956 scope.go:122] "RemoveContainer" containerID="d199e233ab634507148c105e24fb9259576b8a29c7c6aee3ed2e10ef72310f18"
Nov 02 18:30:15.701860 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:15.701820    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerStarted","Data":"b8eaeb1e45508072ce7b2ae0ff4677b154e69a35fe671076644f3c7eb1a4e88b"}
Nov 02 18:30:15.704065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:15.704037    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerStarted","Data":"f6328cae7d4b3ca6844fd9d766c314007c0b4482e1625af2d853af4d742839d7"}
Nov 02 18:30:15.858365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:15.858331    4956 scope.go:122] "RemoveContainer" containerID="b6b5089d470698561960b0b88a8855e21ad8708a43de6496e5f852fc9dc4c027"
Nov 02 18:30:15.985656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:15.985572    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"]
Nov 02 18:30:15.996160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:15.992103    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.044657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.044625    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"]
Nov 02 18:30:16.044888 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.044867    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6cf852\""
Nov 02 18:30:16.044981 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.044961    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6cf852-dockercfg-22g9l\""
Nov 02 18:30:16.094218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.094183    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgwv8\" (UniqueName: \"kubernetes.io/projected/69e6b835-5238-4380-a558-c23cdeb8fbe3-kube-api-access-kgwv8\") pod \"payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.094415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.094381    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6b835-5238-4380-a558-c23cdeb8fbe3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.196345 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.196311    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgwv8\" (UniqueName: \"kubernetes.io/projected/69e6b835-5238-4380-a558-c23cdeb8fbe3-kube-api-access-kgwv8\") pod \"payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.196501 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.196439    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6b835-5238-4380-a558-c23cdeb8fbe3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.197156 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.197135    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6b835-5238-4380-a558-c23cdeb8fbe3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.253591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.253565    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgwv8\" (UniqueName: \"kubernetes.io/projected/69e6b835-5238-4380-a558-c23cdeb8fbe3-kube-api-access-kgwv8\") pod \"payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.311385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.311348    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:16.690079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.690049    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"]
Nov 02 18:30:16.690834 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:30:16.690807    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e6b835_5238_4380_a558_c23cdeb8fbe3.slice/crio-2f1d053d566066a68c78acd83e2a494ab1ee6a70e4f7bd547c3c12b307e6ebc8 WatchSource:0}: Error finding container 2f1d053d566066a68c78acd83e2a494ab1ee6a70e4f7bd547c3c12b307e6ebc8: Status 404 returned error can't find the container with id 2f1d053d566066a68c78acd83e2a494ab1ee6a70e4f7bd547c3c12b307e6ebc8
Nov 02 18:30:16.716132 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.716090    4956 generic.go:356] "Generic (PLEG): container finished" podID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerID="b8eaeb1e45508072ce7b2ae0ff4677b154e69a35fe671076644f3c7eb1a4e88b" exitCode=60
Nov 02 18:30:16.716260 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.716149    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerDied","Data":"b8eaeb1e45508072ce7b2ae0ff4677b154e69a35fe671076644f3c7eb1a4e88b"}
Nov 02 18:30:16.716260 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.716225    4956 scope.go:122] "RemoveContainer" containerID="7f88e0c5ab1d1fa128cfe64c393e4eef7302c07ed6e8a3cbdb6df77efdc2ac32"
Nov 02 18:30:16.716659 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.716638    4956 scope.go:122] "RemoveContainer" containerID="b8eaeb1e45508072ce7b2ae0ff4677b154e69a35fe671076644f3c7eb1a4e88b"
Nov 02 18:30:16.717004 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:16.716980    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm_odh-ai-gateway-infra(6226cfb6-2712-4c01-9d36-baa4e69a749f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f"
Nov 02 18:30:16.718524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.718504    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerStarted","Data":"12f223c83e6c2be77dabcc6a7565fd74ff38582127599188fa2150c0e44aac01"}
Nov 02 18:30:16.719891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.719866    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp" event={"ID":"69e6b835-5238-4380-a558-c23cdeb8fbe3","Type":"ContainerStarted","Data":"2f1d053d566066a68c78acd83e2a494ab1ee6a70e4f7bd547c3c12b307e6ebc8"}
Nov 02 18:30:16.721716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.721694    4956 generic.go:356] "Generic (PLEG): container finished" podID="81694129-39c4-412d-bcaf-ab281c89be8b" containerID="f6328cae7d4b3ca6844fd9d766c314007c0b4482e1625af2d853af4d742839d7" exitCode=60
Nov 02 18:30:16.721814 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.721759    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerDied","Data":"f6328cae7d4b3ca6844fd9d766c314007c0b4482e1625af2d853af4d742839d7"}
Nov 02 18:30:16.722099 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.722084    4956 scope.go:122] "RemoveContainer" containerID="f6328cae7d4b3ca6844fd9d766c314007c0b4482e1625af2d853af4d742839d7"
Nov 02 18:30:16.722337 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:16.722321    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh_odh-ai-gateway-infra(81694129-39c4-412d-bcaf-ab281c89be8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" podUID="81694129-39c4-412d-bcaf-ab281c89be8b"
Nov 02 18:30:16.735356 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.735331    4956 scope.go:122] "RemoveContainer" containerID="d199e233ab634507148c105e24fb9259576b8a29c7c6aee3ed2e10ef72310f18"
Nov 02 18:30:16.828869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.828840    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"]
Nov 02 18:30:16.858060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.858030    4956 scope.go:122] "RemoveContainer" containerID="5779089d23fbcb165e6b892d89f1ea87fbd28321f10b4b5d356e1a1cf2da8756"
Nov 02 18:30:16.949426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:16.949341    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"]
Nov 02 18:30:17.734628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.734595    4956 generic.go:356] "Generic (PLEG): container finished" podID="865e7494-ecdc-4041-a209-35b47628e625" containerID="12f223c83e6c2be77dabcc6a7565fd74ff38582127599188fa2150c0e44aac01" exitCode=60
Nov 02 18:30:17.734792 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.734675    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerDied","Data":"12f223c83e6c2be77dabcc6a7565fd74ff38582127599188fa2150c0e44aac01"}
Nov 02 18:30:17.735071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.735042    4956 scope.go:122] "RemoveContainer" containerID="b6b5089d470698561960b0b88a8855e21ad8708a43de6496e5f852fc9dc4c027"
Nov 02 18:30:17.735537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.735518    4956 scope.go:122] "RemoveContainer" containerID="12f223c83e6c2be77dabcc6a7565fd74ff38582127599188fa2150c0e44aac01"
Nov 02 18:30:17.736382 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:17.735915    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k_odh-ai-gateway-infra(865e7494-ecdc-4041-a209-35b47628e625)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" podUID="865e7494-ecdc-4041-a209-35b47628e625"
Nov 02 18:30:17.738992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.737978    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp" event={"ID":"69e6b835-5238-4380-a558-c23cdeb8fbe3","Type":"ContainerStarted","Data":"2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724"}
Nov 02 18:30:17.738992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.738034    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:17.744978 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.744858    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" event={"ID":"1ee52229-2f22-4de4-973b-df0f5d917cb6","Type":"ContainerStarted","Data":"62eb15fa5bf78fedbcfbf698285a490e7fe2908820ece90a90b41cbcb7672a2a"}
Nov 02 18:30:17.863176 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.857991    4956 scope.go:122] "RemoveContainer" containerID="0897a0c579f518179836c880a50fa1361f3333ee276ca46e759282943171f583"
Nov 02 18:30:17.984345 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.984271    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:17.987348 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:17.987324    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:18.106741 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.106709    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"]
Nov 02 18:30:18.121929 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.121858    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81694129-39c4-412d-bcaf-ab281c89be8b-kube-api-access-ns8tl\" (UniqueName: \"kubernetes.io/projected/81694129-39c4-412d-bcaf-ab281c89be8b-kube-api-access-ns8tl\") pod \"81694129-39c4-412d-bcaf-ab281c89be8b\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") "
Nov 02 18:30:18.122175 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.122040    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6226cfb6-2712-4c01-9d36-baa4e69a749f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6226cfb6-2712-4c01-9d36-baa4e69a749f-maas-api-service-ca\") pod \"6226cfb6-2712-4c01-9d36-baa4e69a749f\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") "
Nov 02 18:30:18.122581 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.122172    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6226cfb6-2712-4c01-9d36-baa4e69a749f-kube-api-access-55pzp\" (UniqueName: \"kubernetes.io/projected/6226cfb6-2712-4c01-9d36-baa4e69a749f-kube-api-access-55pzp\") pod \"6226cfb6-2712-4c01-9d36-baa4e69a749f\" (UID: \"6226cfb6-2712-4c01-9d36-baa4e69a749f\") "
Nov 02 18:30:18.122581 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.122206    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81694129-39c4-412d-bcaf-ab281c89be8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81694129-39c4-412d-bcaf-ab281c89be8b-maas-api-service-ca\") pod \"81694129-39c4-412d-bcaf-ab281c89be8b\" (UID: \"81694129-39c4-412d-bcaf-ab281c89be8b\") "
Nov 02 18:30:18.122904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.122779    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81694129-39c4-412d-bcaf-ab281c89be8b-maas-api-service-ca" pod "81694129-39c4-412d-bcaf-ab281c89be8b" (UID: "81694129-39c4-412d-bcaf-ab281c89be8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:18.122964 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.122933    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81694129-39c4-412d-bcaf-ab281c89be8b-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:18.123584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.122611    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6226cfb6-2712-4c01-9d36-baa4e69a749f-maas-api-service-ca" pod "6226cfb6-2712-4c01-9d36-baa4e69a749f" (UID: "6226cfb6-2712-4c01-9d36-baa4e69a749f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:18.124541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.124509    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81694129-39c4-412d-bcaf-ab281c89be8b-kube-api-access-ns8tl" pod "81694129-39c4-412d-bcaf-ab281c89be8b" (UID: "81694129-39c4-412d-bcaf-ab281c89be8b"). InnerVolumeSpecName "kube-api-access-ns8tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:18.124910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.124890    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6226cfb6-2712-4c01-9d36-baa4e69a749f-kube-api-access-55pzp" pod "6226cfb6-2712-4c01-9d36-baa4e69a749f" (UID: "6226cfb6-2712-4c01-9d36-baa4e69a749f"). InnerVolumeSpecName "kube-api-access-55pzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:18.224055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.224020    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns8tl\" (UniqueName: \"kubernetes.io/projected/81694129-39c4-412d-bcaf-ab281c89be8b-kube-api-access-ns8tl\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:18.224055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.224051    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6226cfb6-2712-4c01-9d36-baa4e69a749f-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:18.224055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.224061    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pzp\" (UniqueName: \"kubernetes.io/projected/6226cfb6-2712-4c01-9d36-baa4e69a749f-kube-api-access-55pzp\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:18.314559 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.314502    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp" podStartSLOduration=3.314489017 podStartE2EDuration="3.314489017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:18.31108738 +0000 UTC m=+4048.568117999" watchObservedRunningTime="2026-11-02 18:30:18.314489017 +0000 UTC m=+4048.571519636"
Nov 02 18:30:18.753642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.753603    4956 generic.go:356] "Generic (PLEG): container finished" podID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerID="62eb15fa5bf78fedbcfbf698285a490e7fe2908820ece90a90b41cbcb7672a2a" exitCode=60
Nov 02 18:30:18.753819 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.753677    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" event={"ID":"1ee52229-2f22-4de4-973b-df0f5d917cb6","Type":"ContainerDied","Data":"62eb15fa5bf78fedbcfbf698285a490e7fe2908820ece90a90b41cbcb7672a2a"}
Nov 02 18:30:18.753819 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.753751    4956 scope.go:122] "RemoveContainer" containerID="5779089d23fbcb165e6b892d89f1ea87fbd28321f10b4b5d356e1a1cf2da8756"
Nov 02 18:30:18.754271 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.754249    4956 scope.go:122] "RemoveContainer" containerID="62eb15fa5bf78fedbcfbf698285a490e7fe2908820ece90a90b41cbcb7672a2a"
Nov 02 18:30:18.754615 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:18.754591    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq_odh-ai-gateway-infra(1ee52229-2f22-4de4-973b-df0f5d917cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6"
Nov 02 18:30:18.756071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.756043    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" event={"ID":"d610f966-10aa-4f0c-a3cc-e16bce11ba4e","Type":"ContainerStarted","Data":"212f5a4b157a40bc688364a33c887c1fea7343c61d1f2183ae671d3c631392f6"}
Nov 02 18:30:18.757401 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.757367    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"
Nov 02 18:30:18.757498 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.757375    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm" event={"ID":"6226cfb6-2712-4c01-9d36-baa4e69a749f","Type":"ContainerDied","Data":"390efe42327a2ffb71510826ca0dbc197aae445ed8034fdabf85708ee66f93a3"}
Nov 02 18:30:18.761083 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.761059    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"
Nov 02 18:30:18.767978 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.767954    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh" event={"ID":"81694129-39c4-412d-bcaf-ab281c89be8b","Type":"ContainerDied","Data":"c0d165c07261603b465782cb9e654c8a227e05073a17b797bc64b149a6d66ed1"}
Nov 02 18:30:18.768096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.768078    4956 scope.go:122] "RemoveContainer" containerID="b8eaeb1e45508072ce7b2ae0ff4677b154e69a35fe671076644f3c7eb1a4e88b"
Nov 02 18:30:18.780409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.780382    4956 scope.go:122] "RemoveContainer" containerID="f6328cae7d4b3ca6844fd9d766c314007c0b4482e1625af2d853af4d742839d7"
Nov 02 18:30:18.902600 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.902573    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:18.979450 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:18.979419    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"]
Nov 02 18:30:19.040597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.040498    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865e7494-ecdc-4041-a209-35b47628e625-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865e7494-ecdc-4041-a209-35b47628e625-maas-api-service-ca\") pod \"865e7494-ecdc-4041-a209-35b47628e625\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") "
Nov 02 18:30:19.041076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.040743    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865e7494-ecdc-4041-a209-35b47628e625-kube-api-access-dzj2t\" (UniqueName: \"kubernetes.io/projected/865e7494-ecdc-4041-a209-35b47628e625-kube-api-access-dzj2t\") pod \"865e7494-ecdc-4041-a209-35b47628e625\" (UID: \"865e7494-ecdc-4041-a209-35b47628e625\") "
Nov 02 18:30:19.041076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.040907    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865e7494-ecdc-4041-a209-35b47628e625-maas-api-service-ca" pod "865e7494-ecdc-4041-a209-35b47628e625" (UID: "865e7494-ecdc-4041-a209-35b47628e625"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:19.041418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.041397    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865e7494-ecdc-4041-a209-35b47628e625-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.043001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.042981    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865e7494-ecdc-4041-a209-35b47628e625-kube-api-access-dzj2t" pod "865e7494-ecdc-4041-a209-35b47628e625" (UID: "865e7494-ecdc-4041-a209-35b47628e625"). InnerVolumeSpecName "kube-api-access-dzj2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:19.142768 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.142723    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzj2t\" (UniqueName: \"kubernetes.io/projected/865e7494-ecdc-4041-a209-35b47628e625-kube-api-access-dzj2t\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.223953 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.223917    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"]
Nov 02 18:30:19.231666 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.231632    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894070-td4cm"]
Nov 02 18:30:19.349303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.349216    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"]
Nov 02 18:30:19.353158 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.353130    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894070-c8cwh"]
Nov 02 18:30:19.772831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.772789    4956 generic.go:356] "Generic (PLEG): container finished" podID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerID="212f5a4b157a40bc688364a33c887c1fea7343c61d1f2183ae671d3c631392f6" exitCode=60
Nov 02 18:30:19.772957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.772819    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" event={"ID":"d610f966-10aa-4f0c-a3cc-e16bce11ba4e","Type":"ContainerDied","Data":"212f5a4b157a40bc688364a33c887c1fea7343c61d1f2183ae671d3c631392f6"}
Nov 02 18:30:19.772957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.772928    4956 scope.go:122] "RemoveContainer" containerID="0897a0c579f518179836c880a50fa1361f3333ee276ca46e759282943171f583"
Nov 02 18:30:19.773371 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.773352    4956 scope.go:122] "RemoveContainer" containerID="212f5a4b157a40bc688364a33c887c1fea7343c61d1f2183ae671d3c631392f6"
Nov 02 18:30:19.773720 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:19.773696    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp_odh-ai-gateway-infra(d610f966-10aa-4f0c-a3cc-e16bce11ba4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e"
Nov 02 18:30:19.775579 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.775557    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k" event={"ID":"865e7494-ecdc-4041-a209-35b47628e625","Type":"ContainerDied","Data":"8eda2f85f2f860ab96906277b5a928b91b81f0d7aef4a4643cdc736bdee06329"}
Nov 02 18:30:19.775670 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.775577    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"
Nov 02 18:30:19.788467 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.788445    4956 scope.go:122] "RemoveContainer" containerID="12f223c83e6c2be77dabcc6a7565fd74ff38582127599188fa2150c0e44aac01"
Nov 02 18:30:19.867212 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.867176    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" path="/var/lib/kubelet/pods/6226cfb6-2712-4c01-9d36-baa4e69a749f/volumes"
Nov 02 18:30:19.867689 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.867670    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" path="/var/lib/kubelet/pods/81694129-39c4-412d-bcaf-ab281c89be8b/volumes"
Nov 02 18:30:19.933705 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:19.933678    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:20.009752 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.009719    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"]
Nov 02 18:30:20.054903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.054814    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee52229-2f22-4de4-973b-df0f5d917cb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee52229-2f22-4de4-973b-df0f5d917cb6-maas-api-service-ca\") pod \"1ee52229-2f22-4de4-973b-df0f5d917cb6\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") "
Nov 02 18:30:20.055288 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.055005    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee52229-2f22-4de4-973b-df0f5d917cb6-kube-api-access-tkhgc\" (UniqueName: \"kubernetes.io/projected/1ee52229-2f22-4de4-973b-df0f5d917cb6-kube-api-access-tkhgc\") pod \"1ee52229-2f22-4de4-973b-df0f5d917cb6\" (UID: \"1ee52229-2f22-4de4-973b-df0f5d917cb6\") "
Nov 02 18:30:20.055288 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.055205    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee52229-2f22-4de4-973b-df0f5d917cb6-maas-api-service-ca" pod "1ee52229-2f22-4de4-973b-df0f5d917cb6" (UID: "1ee52229-2f22-4de4-973b-df0f5d917cb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:20.055462 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.055447    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee52229-2f22-4de4-973b-df0f5d917cb6-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.057304 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.057283    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee52229-2f22-4de4-973b-df0f5d917cb6-kube-api-access-tkhgc" pod "1ee52229-2f22-4de4-973b-df0f5d917cb6" (UID: "1ee52229-2f22-4de4-973b-df0f5d917cb6"). InnerVolumeSpecName "kube-api-access-tkhgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:20.156954 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.156918    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhgc\" (UniqueName: \"kubernetes.io/projected/1ee52229-2f22-4de4-973b-df0f5d917cb6-kube-api-access-tkhgc\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.189367 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.189328    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"]
Nov 02 18:30:20.198151 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.198109    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894070-mnk8k"]
Nov 02 18:30:20.785664 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.785637    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"
Nov 02 18:30:20.785664 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.785654    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq" event={"ID":"1ee52229-2f22-4de4-973b-df0f5d917cb6","Type":"ContainerDied","Data":"59f0f422628eb151a00103ba8ea9a0b66d543c0e45475a8da08ae7897c60ea1d"}
Nov 02 18:30:20.785925 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.785723    4956 scope.go:122] "RemoveContainer" containerID="62eb15fa5bf78fedbcfbf698285a490e7fe2908820ece90a90b41cbcb7672a2a"
Nov 02 18:30:20.947938 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.947913    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:20.952422 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.952396    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"]
Nov 02 18:30:20.955544 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:20.955519    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894070-q9cqq"]
Nov 02 18:30:21.068914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.068831    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-maas-api-service-ca\") pod \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") "
Nov 02 18:30:21.069426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.069067    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-kube-api-access-jhrr2\" (UniqueName: \"kubernetes.io/projected/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-kube-api-access-jhrr2\") pod \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\" (UID: \"d610f966-10aa-4f0c-a3cc-e16bce11ba4e\") "
Nov 02 18:30:21.069426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.069276    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-maas-api-service-ca" pod "d610f966-10aa-4f0c-a3cc-e16bce11ba4e" (UID: "d610f966-10aa-4f0c-a3cc-e16bce11ba4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:21.069515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.069474    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:21.071470 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.071443    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-kube-api-access-jhrr2" pod "d610f966-10aa-4f0c-a3cc-e16bce11ba4e" (UID: "d610f966-10aa-4f0c-a3cc-e16bce11ba4e"). InnerVolumeSpecName "kube-api-access-jhrr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:21.171073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.171040    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrr2\" (UniqueName: \"kubernetes.io/projected/d610f966-10aa-4f0c-a3cc-e16bce11ba4e-kube-api-access-jhrr2\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:21.796938 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.796909    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"
Nov 02 18:30:21.797111 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.796943    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp" event={"ID":"d610f966-10aa-4f0c-a3cc-e16bce11ba4e","Type":"ContainerDied","Data":"376ff6a8412720a4a39f6284c6c8697a51f5792aff92666420c22931779e91d6"}
Nov 02 18:30:21.797111 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.797006    4956 scope.go:122] "RemoveContainer" containerID="212f5a4b157a40bc688364a33c887c1fea7343c61d1f2183ae671d3c631392f6"
Nov 02 18:30:21.867000 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.866964    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" path="/var/lib/kubelet/pods/1ee52229-2f22-4de4-973b-df0f5d917cb6/volumes"
Nov 02 18:30:21.867488 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.867473    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865e7494-ecdc-4041-a209-35b47628e625" path="/var/lib/kubelet/pods/865e7494-ecdc-4041-a209-35b47628e625/volumes"
Nov 02 18:30:21.963958 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.963921    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"]
Nov 02 18:30:21.967343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:21.967313    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894070-fgzmp"]
Nov 02 18:30:22.859641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:22.859605    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:23.865760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:23.865723    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" path="/var/lib/kubelet/pods/d610f966-10aa-4f0c-a3cc-e16bce11ba4e/volumes"
Nov 02 18:30:28.765555 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:28.765525    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:30:29.053959 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:29.053870    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:30:32.859802 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:32.859766    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:42.859259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:42.859221    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:52.859728 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:52.859690    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:53.858523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:30:53.858480    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:59.052760 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:30:59.052717    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:31:03.858697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:03.858616    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:31:09.758454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.758417    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"]
Nov 02 18:31:09.759096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759082    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:09.759165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759098    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:09.759165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759109    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759133    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759155    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:09.759165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759162    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759174    4956 cpu_manager.go:414] "containerMap: removing container" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759181    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759192    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759200    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759215    4956 cpu_manager.go:414] "containerMap: removing container" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759220    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759231    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759235    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759245    4956 cpu_manager.go:414] "containerMap: removing container" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759249    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759254    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759258    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759273    4956 cpu_manager.go:414] "containerMap: removing container" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759277    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759285    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759290    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759300    4956 cpu_manager.go:414] "containerMap: removing container" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759304    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759413    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759423    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759429    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759435    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759441    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759447    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759453    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759462    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759468    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759473    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:09.759909 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.759478    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:09.762993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.762975    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.828412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.818605    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-608907-openshift-default-dockercfg-c6zdr\""
Nov 02 18:31:09.874493 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.874460    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"]
Nov 02 18:31:09.897449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897415    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-socket\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897651 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897529    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-podinfo\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897651 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897609    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-credential-socket\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897651 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897645    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-token\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897782 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897769    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-envoy\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897829 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897816    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-certs\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897849    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnb9m\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-kube-api-access-hnb9m\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897902 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897890    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee971e43-ac12-4c34-892f-a02c57da26e9-istiod-ca-cert\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.897936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.897917    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-data\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999134 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999087    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-envoy\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999178    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-certs\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999222    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnb9m\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-kube-api-access-hnb9m\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999332    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee971e43-ac12-4c34-892f-a02c57da26e9-istiod-ca-cert\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999666 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999634    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-data\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999669    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-certs\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999815    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-socket\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999879    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-podinfo\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999887    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-data\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:09.999975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999948    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-credential-socket\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.000174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:09.999986    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-token\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.000174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.000074    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee971e43-ac12-4c34-892f-a02c57da26e9-istiod-ca-cert\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.000174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.000152    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-socket\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.000430 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.000408    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-credential-socket\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.001812 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.001792    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-envoy\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.002310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.002294    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-podinfo\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.057475 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.057386    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-token\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.057813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.057790    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnb9m\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-kube-api-access-hnb9m\") pod \"e2e-inf-b-608907-openshift-default-6767cf965-rlld4\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") " pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.078039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.078002    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:10.507388 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.507341    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"]
Nov 02 18:31:10.508564 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:31:10.508536    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee971e43_ac12_4c34_892f_a02c57da26e9.slice/crio-72f8b05cdb1377323cec67a55cd592316735636f351258ddc7cc20e305b5d3d8 WatchSource:0}: Error finding container 72f8b05cdb1377323cec67a55cd592316735636f351258ddc7cc20e305b5d3d8: Status 404 returned error can't find the container with id 72f8b05cdb1377323cec67a55cd592316735636f351258ddc7cc20e305b5d3d8
Nov 02 18:31:10.510978 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.510946    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:31:10.511067 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.511010    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:31:10.511067 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:10.511044    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:31:11.134576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:11.134537    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" event={"ID":"ee971e43-ac12-4c34-892f-a02c57da26e9","Type":"ContainerStarted","Data":"b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd"}
Nov 02 18:31:11.135007 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:11.134610    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" event={"ID":"ee971e43-ac12-4c34-892f-a02c57da26e9","Type":"ContainerStarted","Data":"72f8b05cdb1377323cec67a55cd592316735636f351258ddc7cc20e305b5d3d8"}
Nov 02 18:31:11.249867 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:11.249808    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podStartSLOduration=2.249793512 podStartE2EDuration="2.249793512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:11.248971239 +0000 UTC m=+4101.506001861" watchObservedRunningTime="2026-11-02 18:31:11.249793512 +0000 UTC m=+4101.506824132"
Nov 02 18:31:12.078878 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:12.078844    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:12.085599 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:12.085574    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:12.141063 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:12.141032    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:12.142036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:12.142015    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:31:13.859179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:13.859137    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:31:17.901253 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.901217    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"]
Nov 02 18:31:17.902824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.902789    4956 cpu_manager.go:414] "containerMap: removing container" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:17.902824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.902820    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:17.903039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.902864    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:17.903039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.902872    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:17.903039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.902919    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:17.903039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.902927    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d610f966-10aa-4f0c-a3cc-e16bce11ba4e" containerName="cleanup"
Nov 02 18:31:17.906052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.906030    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="81694129-39c4-412d-bcaf-ab281c89be8b" containerName="cleanup"
Nov 02 18:31:17.906052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.906060    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee52229-2f22-4de4-973b-df0f5d917cb6" containerName="cleanup"
Nov 02 18:31:17.906328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.906071    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6226cfb6-2712-4c01-9d36-baa4e69a749f" containerName="cleanup"
Nov 02 18:31:17.906328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.906100    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="865e7494-ecdc-4041-a209-35b47628e625" containerName="cleanup"
Nov 02 18:31:17.911239 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.911212    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:17.955860 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:17.955825    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"]
Nov 02 18:31:18.002323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.002288    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-608907\""
Nov 02 18:31:18.002523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.002349    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-608907-dockercfg-zwrlw\""
Nov 02 18:31:18.006102 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.006077    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdj7x\" (UniqueName: \"kubernetes.io/projected/21e55d78-893c-4f4b-af11-41b397777898-kube-api-access-rdj7x\") pod \"payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") " pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:18.006267 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.006232    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e55d78-893c-4f4b-af11-41b397777898-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") " pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:18.058272 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.058241    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"]
Nov 02 18:31:18.062504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.062488    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.107673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.107641    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e55d78-893c-4f4b-af11-41b397777898-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") " pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:18.107885 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.107713    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.107885 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.107793    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscz6\" (UniqueName: \"kubernetes.io/projected/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-kube-api-access-rscz6\") pod \"payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.108017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.107952    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdj7x\" (UniqueName: \"kubernetes.io/projected/21e55d78-893c-4f4b-af11-41b397777898-kube-api-access-rdj7x\") pod \"payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") " pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:18.108412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.108392    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e55d78-893c-4f4b-af11-41b397777898-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") " pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:18.117984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.117954    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"]
Nov 02 18:31:18.164464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.164389    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdj7x\" (UniqueName: \"kubernetes.io/projected/21e55d78-893c-4f4b-af11-41b397777898-kube-api-access-rdj7x\") pod \"payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") " pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:18.209356 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.209314    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.209552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.209414    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscz6\" (UniqueName: \"kubernetes.io/projected/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-kube-api-access-rscz6\") pod \"payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.210094 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.210073    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.230317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.230281    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:18.266532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.266505    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscz6\" (UniqueName: \"kubernetes.io/projected/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-kube-api-access-rscz6\") pod \"payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.377061 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.377026    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:18.608869 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.608844    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"]
Nov 02 18:31:18.609895 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:31:18.609867    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e55d78_893c_4f4b_af11_41b397777898.slice/crio-d6e4ff6fccc66d4f8b2cfb4ac0b1cd9741be2d2bbe2fed5b69a2f182b6600071 WatchSource:0}: Error finding container d6e4ff6fccc66d4f8b2cfb4ac0b1cd9741be2d2bbe2fed5b69a2f182b6600071: Status 404 returned error can't find the container with id d6e4ff6fccc66d4f8b2cfb4ac0b1cd9741be2d2bbe2fed5b69a2f182b6600071
Nov 02 18:31:18.754952 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:18.754924    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"]
Nov 02 18:31:18.755563 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:31:18.755537    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5530c74a_b5e4_46fe_a7cc_0907e34b1afc.slice/crio-54cd6a9e424f47ba28de74a515bb059fdaabcf75cbd943e644f5e848b9e291a8 WatchSource:0}: Error finding container 54cd6a9e424f47ba28de74a515bb059fdaabcf75cbd943e644f5e848b9e291a8: Status 404 returned error can't find the container with id 54cd6a9e424f47ba28de74a515bb059fdaabcf75cbd943e644f5e848b9e291a8
Nov 02 18:31:19.188800 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:19.188711    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt" event={"ID":"5530c74a-b5e4-46fe-a7cc-0907e34b1afc","Type":"ContainerStarted","Data":"9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68"}
Nov 02 18:31:19.188800 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:19.188789    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:19.189312 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:19.188819    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt" event={"ID":"5530c74a-b5e4-46fe-a7cc-0907e34b1afc","Type":"ContainerStarted","Data":"54cd6a9e424f47ba28de74a515bb059fdaabcf75cbd943e644f5e848b9e291a8"}
Nov 02 18:31:19.190729 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:19.190700    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t" event={"ID":"21e55d78-893c-4f4b-af11-41b397777898","Type":"ContainerStarted","Data":"7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188"}
Nov 02 18:31:19.190883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:19.190752    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t" event={"ID":"21e55d78-893c-4f4b-af11-41b397777898","Type":"ContainerStarted","Data":"d6e4ff6fccc66d4f8b2cfb4ac0b1cd9741be2d2bbe2fed5b69a2f182b6600071"}
Nov 02 18:31:19.190883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:19.190785    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:19.302850 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:19.302804    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt" podStartSLOduration=1.30279089 podStartE2EDuration="1.30279089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:19.30133922 +0000 UTC m=+4109.558369853" watchObservedRunningTime="2026-11-02 18:31:19.30279089 +0000 UTC m=+4109.559821521"
Nov 02 18:31:23.858665 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:23.858624    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:31:29.054599 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:31:29.054559    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:31:30.201990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:30.201957    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:31:30.202433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:30.202039    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:31:30.317255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:30.317204    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t" podStartSLOduration=13.317190064 podStartE2EDuration="13.317190064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:19.424376768 +0000 UTC m=+4109.681407389" watchObservedRunningTime="2026-11-02 18:31:30.317190064 +0000 UTC m=+4120.574220684"
Nov 02 18:31:33.858531 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:33.858490    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:31:43.859036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:43.858995    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:31:51.667978 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:31:51.667943    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:31:51.667978 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:31:51.667971    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:31:51.668536 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:31:51.668037    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:33:53.668018169 +0000 UTC m=+4263.925048786 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:31:53.858641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:53.858600    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:31:57.858645 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:31:57.858602    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:31:59.053403 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:31:59.053359    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:32:07.858609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:07.858564    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:32:17.858727 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:17.858687    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:32:27.858806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:27.858761    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:32:29.053907 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:32:29.053870    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:32:37.858669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:37.858589    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:32:47.858880 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:47.858839    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:32:50.485010 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:50.484981    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:32:50.489367 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:50.489339    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:32:57.858681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:32:57.858640    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:32:59.053340 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:32:59.053303    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:33:04.859074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:04.859035    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:33:13.183309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.183272    4956 patch_prober.go:29] interesting pod/e2e-inf-b-608907-openshift-default-6767cf965-rlld4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:33:13.183713 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.183354    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:33:13.996769 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.996738    4956 generic.go:356] "Generic (PLEG): container finished" podID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerID="b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd" exitCode=0
Nov 02 18:33:13.997194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.996811    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" event={"ID":"ee971e43-ac12-4c34-892f-a02c57da26e9","Type":"ContainerDied","Data":"b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd"}
Nov 02 18:33:13.997373 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.997356    4956 scope.go:122] "RemoveContainer" containerID="b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd"
Nov 02 18:33:13.999077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.999049    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:33:13.999233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.999150    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:33:13.999233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:13.999185    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:33:14.859167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:14.859094    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:33:15.005465 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:15.005432    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" event={"ID":"ee971e43-ac12-4c34-892f-a02c57da26e9","Type":"ContainerStarted","Data":"5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9"}
Nov 02 18:33:15.079083 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:15.079042    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:33:15.079339 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:15.079313    4956 patch_prober.go:29] interesting pod/e2e-inf-b-608907-openshift-default-6767cf965-rlld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 02 18:33:15.079454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:15.079382    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 02 18:33:16.079299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:16.079264    4956 patch_prober.go:29] interesting pod/e2e-inf-b-608907-openshift-default-6767cf965-rlld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 02 18:33:16.079722 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:16.079315    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 02 18:33:17.079369 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:17.079333    4956 patch_prober.go:29] interesting pod/e2e-inf-b-608907-openshift-default-6767cf965-rlld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 02 18:33:17.079823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:17.079407    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 02 18:33:18.079090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:18.079048    4956 patch_prober.go:29] interesting pod/e2e-inf-b-608907-openshift-default-6767cf965-rlld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 02 18:33:18.079328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:18.079142    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 02 18:33:19.078471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:19.078436    4956 patch_prober.go:29] interesting pod/e2e-inf-b-608907-openshift-default-6767cf965-rlld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 02 18:33:19.078898 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:19.078498    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 02 18:33:20.079063 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:20.079024    4956 patch_prober.go:29] interesting pod/e2e-inf-b-608907-openshift-default-6767cf965-rlld4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 02 18:33:20.079620 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:20.079101    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 02 18:33:21.084257 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:21.084225    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:33:21.084744 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:21.084704    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:33:21.085632 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:21.085610    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:33:22.218653 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:22.217911    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"]
Nov 02 18:33:24.858848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:24.858811    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:33:27.218725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.218678    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz" podUID="321bd38a-f405-47e5-b9a7-52c6fe4df243" containerName="payload-pre-processing" containerID="cri-o://66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9" gracePeriod=25
Nov 02 18:33:27.513626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.513599    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:33:27.577603 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.577569    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321bd38a-f405-47e5-b9a7-52c6fe4df243-kube-api-access-v9cgd\" (UniqueName: \"kubernetes.io/projected/321bd38a-f405-47e5-b9a7-52c6fe4df243-kube-api-access-v9cgd\") pod \"321bd38a-f405-47e5-b9a7-52c6fe4df243\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") "
Nov 02 18:33:27.577811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.577688    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321bd38a-f405-47e5-b9a7-52c6fe4df243-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321bd38a-f405-47e5-b9a7-52c6fe4df243-plugins-config-volume\") pod \"321bd38a-f405-47e5-b9a7-52c6fe4df243\" (UID: \"321bd38a-f405-47e5-b9a7-52c6fe4df243\") "
Nov 02 18:33:27.578074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.578049    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321bd38a-f405-47e5-b9a7-52c6fe4df243-plugins-config-volume" pod "321bd38a-f405-47e5-b9a7-52c6fe4df243" (UID: "321bd38a-f405-47e5-b9a7-52c6fe4df243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:27.578286 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.578269    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321bd38a-f405-47e5-b9a7-52c6fe4df243-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.579976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.579949    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321bd38a-f405-47e5-b9a7-52c6fe4df243-kube-api-access-v9cgd" pod "321bd38a-f405-47e5-b9a7-52c6fe4df243" (UID: "321bd38a-f405-47e5-b9a7-52c6fe4df243"). InnerVolumeSpecName "kube-api-access-v9cgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:27.679749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:27.679718    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cgd\" (UniqueName: \"kubernetes.io/projected/321bd38a-f405-47e5-b9a7-52c6fe4df243-kube-api-access-v9cgd\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:28.124960 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.124927    4956 generic.go:356] "Generic (PLEG): container finished" podID="321bd38a-f405-47e5-b9a7-52c6fe4df243" containerID="66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9" exitCode=0
Nov 02 18:33:28.124960 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.124971    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"
Nov 02 18:33:28.125252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.125006    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz" event={"ID":"321bd38a-f405-47e5-b9a7-52c6fe4df243","Type":"ContainerDied","Data":"66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9"}
Nov 02 18:33:28.125252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.125060    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz" event={"ID":"321bd38a-f405-47e5-b9a7-52c6fe4df243","Type":"ContainerDied","Data":"b3883ccd1f41497f9b12df6bff3321a5c852ea7e3decf8400588f1829dbdc1bd"}
Nov 02 18:33:28.125252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.125085    4956 scope.go:122] "RemoveContainer" containerID="66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9"
Nov 02 18:33:28.138364 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.138337    4956 scope.go:122] "RemoveContainer" containerID="66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9"
Nov 02 18:33:28.138686 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:33:28.138664    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9\": container with ID starting with 66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9 not found: ID does not exist" containerID="66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9"
Nov 02 18:33:28.138765 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.138694    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9"} err="failed to get container status \"66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9\": rpc error: code = NotFound desc = could not find container \"66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9\": container with ID starting with 66268b460c0c90207544a36cfa5ccae7b028923041399d10b7f420c5fc201bc9 not found: ID does not exist"
Nov 02 18:33:28.288094 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.288062    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"]
Nov 02 18:33:28.291573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:28.291548    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-662a80-9cd9d57ff-cg6vz"]
Nov 02 18:33:29.055820 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:33:29.055782    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:33:29.865906 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:29.865869    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321bd38a-f405-47e5-b9a7-52c6fe4df243" path="/var/lib/kubelet/pods/321bd38a-f405-47e5-b9a7-52c6fe4df243/volumes"
Nov 02 18:33:34.859034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:34.858998    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:33:44.858910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:44.858873    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:33:49.112050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.112016    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"]
Nov 02 18:33:49.216804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.216769    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"]
Nov 02 18:33:49.700628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.700592    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"]
Nov 02 18:33:49.701383 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.701365    4956 cpu_manager.go:414] "containerMap: removing container" podUID="321bd38a-f405-47e5-b9a7-52c6fe4df243" containerName="payload-pre-processing"
Nov 02 18:33:49.701476 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.701384    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321bd38a-f405-47e5-b9a7-52c6fe4df243" containerName="payload-pre-processing"
Nov 02 18:33:49.701546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.701500    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="321bd38a-f405-47e5-b9a7-52c6fe4df243" containerName="payload-pre-processing"
Nov 02 18:33:49.705299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.705276    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.757167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.757098    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:49.757464 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.757206    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:33:49.759760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.759734    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"]
Nov 02 18:33:49.771280 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.771247    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrsz\" (UniqueName: \"kubernetes.io/projected/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-kube-api-access-jgrsz\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.771479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.771384    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.771568 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.771549    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.872992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.872957    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.873205 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.873072    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrsz\" (UniqueName: \"kubernetes.io/projected/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-kube-api-access-jgrsz\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.873205 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.873147    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.873787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.873760    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.923662 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.923623    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:49.929535 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.929505    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrsz\" (UniqueName: \"kubernetes.io/projected/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-kube-api-access-jgrsz\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:49.935758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:49.935732    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67abda02-b4lp8\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:50.068782 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:50.068703    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:33:50.078939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:50.078909    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:50.455754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:50.455730    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"]
Nov 02 18:33:50.457151 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:33:50.457091    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f81d94f_0c20_436a_8df7_3aa4e6f121ab.slice/crio-3c869cf89575655f5b0e5e21ecf2c2930e0763bab422e2eacae3e95e8b4ccded WatchSource:0}: Error finding container 3c869cf89575655f5b0e5e21ecf2c2930e0763bab422e2eacae3e95e8b4ccded: Status 404 returned error can't find the container with id 3c869cf89575655f5b0e5e21ecf2c2930e0763bab422e2eacae3e95e8b4ccded
Nov 02 18:33:51.289573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:51.289485    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8" event={"ID":"3f81d94f-0c20-436a-8df7-3aa4e6f121ab","Type":"ContainerStarted","Data":"16226edbaf47551296f71710632882607a25594c0842ee1cd66c557257605305"}
Nov 02 18:33:51.289573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:51.289541    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8" event={"ID":"3f81d94f-0c20-436a-8df7-3aa4e6f121ab","Type":"ContainerStarted","Data":"3c869cf89575655f5b0e5e21ecf2c2930e0763bab422e2eacae3e95e8b4ccded"}
Nov 02 18:33:51.403680 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:51.403629    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8" podStartSLOduration=2.403615006 podStartE2EDuration="2.403615006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:33:51.400649739 +0000 UTC m=+4261.657680361" watchObservedRunningTime="2026-11-02 18:33:51.403615006 +0000 UTC m=+4261.660645625"
Nov 02 18:33:52.297233 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:52.297193    4956 generic.go:356] "Generic (PLEG): container finished" podID="3f81d94f-0c20-436a-8df7-3aa4e6f121ab" containerID="16226edbaf47551296f71710632882607a25594c0842ee1cd66c557257605305" exitCode=0
Nov 02 18:33:52.297650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:52.297285    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8" event={"ID":"3f81d94f-0c20-436a-8df7-3aa4e6f121ab","Type":"ContainerDied","Data":"16226edbaf47551296f71710632882607a25594c0842ee1cd66c557257605305"}
Nov 02 18:33:53.449916 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.449882    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:53.519936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.519906    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-openshift-ca\") pod \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") "
Nov 02 18:33:53.520152 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.519974    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-xks-ca\") pod \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") "
Nov 02 18:33:53.520299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.520278    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-kube-api-access-jgrsz\" (UniqueName: \"kubernetes.io/projected/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-kube-api-access-jgrsz\") pod \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\" (UID: \"3f81d94f-0c20-436a-8df7-3aa4e6f121ab\") "
Nov 02 18:33:53.520365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.520347    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-openshift-ca" pod "3f81d94f-0c20-436a-8df7-3aa4e6f121ab" (UID: "3f81d94f-0c20-436a-8df7-3aa4e6f121ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:53.520931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.520868    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-openshift-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:53.522214 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.522189    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-xks-ca" pod "3f81d94f-0c20-436a-8df7-3aa4e6f121ab" (UID: "3f81d94f-0c20-436a-8df7-3aa4e6f121ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:53.522449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.522427    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-kube-api-access-jgrsz" pod "3f81d94f-0c20-436a-8df7-3aa4e6f121ab" (UID: "3f81d94f-0c20-436a-8df7-3aa4e6f121ab"). InnerVolumeSpecName "kube-api-access-jgrsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:53.622894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.622808    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrsz\" (UniqueName: \"kubernetes.io/projected/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-kube-api-access-jgrsz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:53.622894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:53.622838    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81d94f-0c20-436a-8df7-3aa4e6f121ab-maas-api-xks-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:53.724257 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:33:53.724220    4956 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ad0e4981cd4f: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:33:53.724257 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:33:53.724247    4956 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w: secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:33:53.724512 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:33:53.724302    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials podName:cd9a4a34-abee-461e-a01a-3f1f89ba22eb nodeName:}" failed. No retries permitted until 2026-11-02 18:35:55.724287861 +0000 UTC m=+4385.981318459 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials") pod "payload-processing-external-model-8ddf7bb45-qzn7w" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb") : secret "agc-cred-ad0e4981cd4f" not found
Nov 02 18:33:54.112584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.112541    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm" podUID="9e1212eb-9fce-4603-b954-32c1b28fbc6a" containerName="payload-processing" containerID="cri-o://dbe2b4e943857f1e8b0955bc2eb32c705cc4c6f3555c0dff1fe2cd70516f28e7" gracePeriod=25
Nov 02 18:33:54.217354 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.217317    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h" podUID="b0ca09da-36ff-4a0f-b87b-948bb48eb768" containerName="payload-pre-processing" containerID="cri-o://b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e" gracePeriod=25
Nov 02 18:33:54.317017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.316986    4956 generic.go:356] "Generic (PLEG): container finished" podID="9e1212eb-9fce-4603-b954-32c1b28fbc6a" containerID="dbe2b4e943857f1e8b0955bc2eb32c705cc4c6f3555c0dff1fe2cd70516f28e7" exitCode=0
Nov 02 18:33:54.317232 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.317065    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm" event={"ID":"9e1212eb-9fce-4603-b954-32c1b28fbc6a","Type":"ContainerDied","Data":"dbe2b4e943857f1e8b0955bc2eb32c705cc4c6f3555c0dff1fe2cd70516f28e7"}
Nov 02 18:33:54.318804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.318780    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8" event={"ID":"3f81d94f-0c20-436a-8df7-3aa4e6f121ab","Type":"ContainerDied","Data":"3c869cf89575655f5b0e5e21ecf2c2930e0763bab422e2eacae3e95e8b4ccded"}
Nov 02 18:33:54.318918 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.318820    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"
Nov 02 18:33:54.318918 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.318826    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c869cf89575655f5b0e5e21ecf2c2930e0763bab422e2eacae3e95e8b4ccded"
Nov 02 18:33:54.387484 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.387458    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:33:54.501672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.501631    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:33:54.535602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.535560    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1212eb-9fce-4603-b954-32c1b28fbc6a-kube-api-access-jkp6r\" (UniqueName: \"kubernetes.io/projected/9e1212eb-9fce-4603-b954-32c1b28fbc6a-kube-api-access-jkp6r\") pod \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") "
Nov 02 18:33:54.535803 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.535709    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1212eb-9fce-4603-b954-32c1b28fbc6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1212eb-9fce-4603-b954-32c1b28fbc6a-plugins-config-volume\") pod \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\" (UID: \"9e1212eb-9fce-4603-b954-32c1b28fbc6a\") "
Nov 02 18:33:54.536166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.536134    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1212eb-9fce-4603-b954-32c1b28fbc6a-plugins-config-volume" pod "9e1212eb-9fce-4603-b954-32c1b28fbc6a" (UID: "9e1212eb-9fce-4603-b954-32c1b28fbc6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:54.536445 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.536352    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1212eb-9fce-4603-b954-32c1b28fbc6a-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:54.538044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.538014    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1212eb-9fce-4603-b954-32c1b28fbc6a-kube-api-access-jkp6r" pod "9e1212eb-9fce-4603-b954-32c1b28fbc6a" (UID: "9e1212eb-9fce-4603-b954-32c1b28fbc6a"). InnerVolumeSpecName "kube-api-access-jkp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:54.637294 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.637207    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ca09da-36ff-4a0f-b87b-948bb48eb768-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ca09da-36ff-4a0f-b87b-948bb48eb768-plugins-config-volume\") pod \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") "
Nov 02 18:33:54.637453 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.637423    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ca09da-36ff-4a0f-b87b-948bb48eb768-kube-api-access-z2hqb\" (UniqueName: \"kubernetes.io/projected/b0ca09da-36ff-4a0f-b87b-948bb48eb768-kube-api-access-z2hqb\") pod \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\" (UID: \"b0ca09da-36ff-4a0f-b87b-948bb48eb768\") "
Nov 02 18:33:54.637592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.637565    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ca09da-36ff-4a0f-b87b-948bb48eb768-plugins-config-volume" pod "b0ca09da-36ff-4a0f-b87b-948bb48eb768" (UID: "b0ca09da-36ff-4a0f-b87b-948bb48eb768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:54.638019 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.637999    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkp6r\" (UniqueName: \"kubernetes.io/projected/9e1212eb-9fce-4603-b954-32c1b28fbc6a-kube-api-access-jkp6r\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:54.638096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.638025    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ca09da-36ff-4a0f-b87b-948bb48eb768-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:54.639785 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.639761    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ca09da-36ff-4a0f-b87b-948bb48eb768-kube-api-access-z2hqb" pod "b0ca09da-36ff-4a0f-b87b-948bb48eb768" (UID: "b0ca09da-36ff-4a0f-b87b-948bb48eb768"). InnerVolumeSpecName "kube-api-access-z2hqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:54.739899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.739866    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2hqb\" (UniqueName: \"kubernetes.io/projected/b0ca09da-36ff-4a0f-b87b-948bb48eb768-kube-api-access-z2hqb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:54.859776 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:54.859731    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:33:55.327130 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.327092    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"
Nov 02 18:33:55.327327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.327147    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm" event={"ID":"9e1212eb-9fce-4603-b954-32c1b28fbc6a","Type":"ContainerDied","Data":"06bb80a440b38cc3fd7788acc6052afbde449c28ae498feef784a6cb7a1bfbd9"}
Nov 02 18:33:55.327327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.327211    4956 scope.go:122] "RemoveContainer" containerID="dbe2b4e943857f1e8b0955bc2eb32c705cc4c6f3555c0dff1fe2cd70516f28e7"
Nov 02 18:33:55.328712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.328689    4956 generic.go:356] "Generic (PLEG): container finished" podID="b0ca09da-36ff-4a0f-b87b-948bb48eb768" containerID="b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e" exitCode=0
Nov 02 18:33:55.328865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.328745    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"
Nov 02 18:33:55.328865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.328762    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h" event={"ID":"b0ca09da-36ff-4a0f-b87b-948bb48eb768","Type":"ContainerDied","Data":"b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e"}
Nov 02 18:33:55.328865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.328798    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h" event={"ID":"b0ca09da-36ff-4a0f-b87b-948bb48eb768","Type":"ContainerDied","Data":"a7bb063d9f51c377b04ec2bd32256f49c866c4cb4773f2419e8c1e9a6339cd45"}
Nov 02 18:33:55.339878 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.339741    4956 scope.go:122] "RemoveContainer" containerID="b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e"
Nov 02 18:33:55.349651 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.349633    4956 scope.go:122] "RemoveContainer" containerID="b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e"
Nov 02 18:33:55.349908 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:33:55.349885    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e\": container with ID starting with b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e not found: ID does not exist" containerID="b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e"
Nov 02 18:33:55.349982 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.349916    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e"} err="failed to get container status \"b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e\": rpc error: code = NotFound desc = could not find container \"b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e\": container with ID starting with b99f934d26940878fefac3d11638d56b47845b453665139a72ba916f2f12e58e not found: ID does not exist"
Nov 02 18:33:55.490369 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.490330    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"]
Nov 02 18:33:55.541343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.541308    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-546881-849d945fc-2m9vm"]
Nov 02 18:33:55.605026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.604942    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"]
Nov 02 18:33:55.607976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.607953    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-546881-65d97ff679-md47h"]
Nov 02 18:33:55.869481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.869404    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1212eb-9fce-4603-b954-32c1b28fbc6a" path="/var/lib/kubelet/pods/9e1212eb-9fce-4603-b954-32c1b28fbc6a/volumes"
Nov 02 18:33:55.869821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:55.869807    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ca09da-36ff-4a0f-b87b-948bb48eb768" path="/var/lib/kubelet/pods/b0ca09da-36ff-4a0f-b87b-948bb48eb768/volumes"
Nov 02 18:33:56.054908 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.054878    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"]
Nov 02 18:33:56.055618 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055595    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b0ca09da-36ff-4a0f-b87b-948bb48eb768" containerName="payload-pre-processing"
Nov 02 18:33:56.055618 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055615    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ca09da-36ff-4a0f-b87b-948bb48eb768" containerName="payload-pre-processing"
Nov 02 18:33:56.055701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055634    4956 cpu_manager.go:414] "containerMap: removing container" podUID="9e1212eb-9fce-4603-b954-32c1b28fbc6a" containerName="payload-processing"
Nov 02 18:33:56.055701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055639    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1212eb-9fce-4603-b954-32c1b28fbc6a" containerName="payload-processing"
Nov 02 18:33:56.055701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055664    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3f81d94f-0c20-436a-8df7-3aa4e6f121ab" containerName="revoke-keys"
Nov 02 18:33:56.055701 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055669    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f81d94f-0c20-436a-8df7-3aa4e6f121ab" containerName="revoke-keys"
Nov 02 18:33:56.055822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055764    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ca09da-36ff-4a0f-b87b-948bb48eb768" containerName="payload-pre-processing"
Nov 02 18:33:56.055822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055774    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1212eb-9fce-4603-b954-32c1b28fbc6a" containerName="payload-processing"
Nov 02 18:33:56.055822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.055787    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f81d94f-0c20-436a-8df7-3aa4e6f121ab" containerName="revoke-keys"
Nov 02 18:33:56.059905 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.059886    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.112688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.112587    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:56.112913 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.112766    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:33:56.116300 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.116277    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"]
Nov 02 18:33:56.158228 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.158146    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.158228 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.158212    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.158425 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.158361    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcff\" (UniqueName: \"kubernetes.io/projected/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-kube-api-access-zqcff\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.259093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.259061    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.259302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.259147    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.259302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.259256    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcff\" (UniqueName: \"kubernetes.io/projected/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-kube-api-access-zqcff\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.259726 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.259706    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.261681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.261660    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.315990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.315959    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcff\" (UniqueName: \"kubernetes.io/projected/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-kube-api-access-zqcff\") pod \"maas-api-revoke-keys-e2e-inf-a-c70712-fsq65\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.373308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.373281    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:56.747592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:56.747567    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"]
Nov 02 18:33:56.749220 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:33:56.749191    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db4aee1_079a_44f5_b43f_2d5ef6a56fb9.slice/crio-cadb506723d0bcc634915e2e63ff14aaccc58d73669b231463e9e87208fd7d78 WatchSource:0}: Error finding container cadb506723d0bcc634915e2e63ff14aaccc58d73669b231463e9e87208fd7d78: Status 404 returned error can't find the container with id cadb506723d0bcc634915e2e63ff14aaccc58d73669b231463e9e87208fd7d78
Nov 02 18:33:57.348012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:57.347977    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65" event={"ID":"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9","Type":"ContainerStarted","Data":"34bef367bedd2334570abfeff8fabf731af923ea9866213b23f8ac7ebf64f074"}
Nov 02 18:33:57.348219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:57.348039    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65" event={"ID":"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9","Type":"ContainerStarted","Data":"cadb506723d0bcc634915e2e63ff14aaccc58d73669b231463e9e87208fd7d78"}
Nov 02 18:33:57.460589 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:57.460314    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65" podStartSLOduration=1.460295346 podStartE2EDuration="1.460295346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:33:57.45924653 +0000 UTC m=+4267.716277151" watchObservedRunningTime="2026-11-02 18:33:57.460295346 +0000 UTC m=+4267.717326001"
Nov 02 18:33:58.356712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:58.356676    4956 generic.go:356] "Generic (PLEG): container finished" podID="1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" containerID="34bef367bedd2334570abfeff8fabf731af923ea9866213b23f8ac7ebf64f074" exitCode=0
Nov 02 18:33:58.357151 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:58.356763    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65" event={"ID":"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9","Type":"ContainerDied","Data":"34bef367bedd2334570abfeff8fabf731af923ea9866213b23f8ac7ebf64f074"}
Nov 02 18:33:58.943747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:58.943705    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"]
Nov 02 18:33:58.943988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:58.943961    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy" containerID="cri-o://29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481" gracePeriod=30
Nov 02 18:33:59.058433 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:33:59.058396    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:33:59.532614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.532549    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:33:59.615423 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.615344    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-xks-ca\") pod \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") "
Nov 02 18:33:59.615423 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.615388    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-kube-api-access-zqcff\" (UniqueName: \"kubernetes.io/projected/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-kube-api-access-zqcff\") pod \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") "
Nov 02 18:33:59.615628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.615521    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-openshift-ca\") pod \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\" (UID: \"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9\") "
Nov 02 18:33:59.615917 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.615881    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-openshift-ca" pod "1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" (UID: "1db4aee1-079a-44f5-b43f-2d5ef6a56fb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:59.616327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.616304    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-openshift-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:59.617546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.617520    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-xks-ca" pod "1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" (UID: "1db4aee1-079a-44f5-b43f-2d5ef6a56fb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:59.617739 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.617715    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-kube-api-access-zqcff" pod "1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" (UID: "1db4aee1-079a-44f5-b43f-2d5ef6a56fb9"). InnerVolumeSpecName "kube-api-access-zqcff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:59.717694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.717659    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-maas-api-xks-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:33:59.717694 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:33:59.717688    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcff\" (UniqueName: \"kubernetes.io/projected/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9-kube-api-access-zqcff\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.371617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:00.371575    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65" event={"ID":"1db4aee1-079a-44f5-b43f-2d5ef6a56fb9","Type":"ContainerDied","Data":"cadb506723d0bcc634915e2e63ff14aaccc58d73669b231463e9e87208fd7d78"}
Nov 02 18:34:00.371797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:00.371637    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cadb506723d0bcc634915e2e63ff14aaccc58d73669b231463e9e87208fd7d78"
Nov 02 18:34:00.371797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:00.371585    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"
Nov 02 18:34:01.978963 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:01.978931    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"]
Nov 02 18:34:01.980463 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:01.980434    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:34:02.028659 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:02.028620    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"]
Nov 02 18:34:04.217206 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.217135    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:34:04.278423 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278383    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-podinfo\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.278629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278434    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-data\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.278629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278472    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-credential-socket\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.278629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278555    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-kube-api-access-5vwcd\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-kube-api-access-5vwcd\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.278805 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278767    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-certs\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.278890 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278825    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-token\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-token\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.278890 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278767    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-data" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:04.278890 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278789    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-credential-socket" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:04.279044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278897    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-envoy\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.279044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278964    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-certs" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:04.279044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.278980    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-socket\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.279044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.279019    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ed5b33-070f-49e9-a5c2-635c0b220827-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ed5b33-070f-49e9-a5c2-635c0b220827-istiod-ca-cert\") pod \"78ed5b33-070f-49e9-a5c2-635c0b220827\" (UID: \"78ed5b33-070f-49e9-a5c2-635c0b220827\") "
Nov 02 18:34:04.279253 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.279224    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-socket" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:04.279842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.279666    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.279842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.279696    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.279842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.279712    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.279842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.279728    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.279842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.279653    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ed5b33-070f-49e9-a5c2-635c0b220827-istiod-ca-cert" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:04.281143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.281099    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-podinfo" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:34:04.281386 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.281367    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-kube-api-access-5vwcd" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "kube-api-access-5vwcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:04.281460 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.281392    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-token" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:04.281519 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.281510    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-envoy" pod "78ed5b33-070f-49e9-a5c2-635c0b220827" (UID: "78ed5b33-070f-49e9-a5c2-635c0b220827"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:04.380820 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.380786    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.380820 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.380816    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vwcd\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-kube-api-access-5vwcd\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.380820 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.380826    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.380820 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.380836    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ed5b33-070f-49e9-a5c2-635c0b220827-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.381112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.380845    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ed5b33-070f-49e9-a5c2-635c0b220827-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:04.403410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.403370    4956 generic.go:356] "Generic (PLEG): container finished" podID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerID="29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481" exitCode=0
Nov 02 18:34:04.403410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.403436    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"
Nov 02 18:34:04.403410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.403460    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" event={"ID":"78ed5b33-070f-49e9-a5c2-635c0b220827","Type":"ContainerDied","Data":"29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481"}
Nov 02 18:34:04.403912 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.403887    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv" event={"ID":"78ed5b33-070f-49e9-a5c2-635c0b220827","Type":"ContainerDied","Data":"770afe19f03599f8cc27ddbee3512610d610b624061a50c3fb5a88eb7ebf8940"}
Nov 02 18:34:04.404053 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.403947    4956 scope.go:122] "RemoveContainer" containerID="29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481"
Nov 02 18:34:04.417731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.417711    4956 scope.go:122] "RemoveContainer" containerID="29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481"
Nov 02 18:34:04.418085 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:34:04.418062    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481\": container with ID starting with 29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481 not found: ID does not exist" containerID="29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481"
Nov 02 18:34:04.418208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.418094    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481"} err="failed to get container status \"29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481\": rpc error: code = NotFound desc = could not find container \"29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481\": container with ID starting with 29b72da61e72cb12d315a793af682a4296b26b782a45191292f5e15dedfdd481 not found: ID does not exist"
Nov 02 18:34:04.567938 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.567848    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"]
Nov 02 18:34:04.576475 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:04.576445    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-prpkv"]
Nov 02 18:34:05.866183 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:05.866149    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" path="/var/lib/kubelet/pods/78ed5b33-070f-49e9-a5c2-635c0b220827/volumes"
Nov 02 18:34:06.979472 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:06.979431    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing" containerID="cri-o://68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644" gracePeriod=25
Nov 02 18:34:07.029217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.029166    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" podUID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" containerName="payload-processing" containerID="cri-o://e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464" gracePeriod=25
Nov 02 18:34:07.255685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.255623    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:34:07.322809 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.322774    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-provider-credentials\") pod \"07985da9-31f1-4365-8a90-9be674b0e6ab\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") "
Nov 02 18:34:07.322809 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.322812    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-routing-overlay-volume\") pod \"07985da9-31f1-4365-8a90-9be674b0e6ab\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") "
Nov 02 18:34:07.323029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.322970    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-plugins-config-volume\") pod \"07985da9-31f1-4365-8a90-9be674b0e6ab\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") "
Nov 02 18:34:07.323086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.323032    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-kube-api-access-tn7cb\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-kube-api-access-tn7cb\") pod \"07985da9-31f1-4365-8a90-9be674b0e6ab\" (UID: \"07985da9-31f1-4365-8a90-9be674b0e6ab\") "
Nov 02 18:34:07.323279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.323252    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-routing-overlay-volume" pod "07985da9-31f1-4365-8a90-9be674b0e6ab" (UID: "07985da9-31f1-4365-8a90-9be674b0e6ab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:07.323440 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.323415    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-plugins-config-volume" pod "07985da9-31f1-4365-8a90-9be674b0e6ab" (UID: "07985da9-31f1-4365-8a90-9be674b0e6ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:07.323578 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.323565    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:07.323621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.323580    4956 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07985da9-31f1-4365-8a90-9be674b0e6ab-routing-overlay-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:07.325301 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.325280    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-provider-credentials" pod "07985da9-31f1-4365-8a90-9be674b0e6ab" (UID: "07985da9-31f1-4365-8a90-9be674b0e6ab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:07.325362 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.325343    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-kube-api-access-tn7cb" pod "07985da9-31f1-4365-8a90-9be674b0e6ab" (UID: "07985da9-31f1-4365-8a90-9be674b0e6ab"). InnerVolumeSpecName "kube-api-access-tn7cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:07.424849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.424808    4956 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-provider-credentials\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:07.424849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.424845    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn7cb\" (UniqueName: \"kubernetes.io/projected/07985da9-31f1-4365-8a90-9be674b0e6ab-kube-api-access-tn7cb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:07.425518 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.425495    4956 generic.go:356] "Generic (PLEG): container finished" podID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerID="68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644" exitCode=0
Nov 02 18:34:07.425587 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.425538    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"
Nov 02 18:34:07.425652 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.425582    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" event={"ID":"07985da9-31f1-4365-8a90-9be674b0e6ab","Type":"ContainerDied","Data":"68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644"}
Nov 02 18:34:07.425709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.425678    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x" event={"ID":"07985da9-31f1-4365-8a90-9be674b0e6ab","Type":"ContainerDied","Data":"6d4d0728c5b5de4e46fc5a030f6b5937662a6951a1156b4b8a27cf67e4488466"}
Nov 02 18:34:07.425770 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.425715    4956 scope.go:122] "RemoveContainer" containerID="68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644"
Nov 02 18:34:07.440054 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.440021    4956 scope.go:122] "RemoveContainer" containerID="68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644"
Nov 02 18:34:07.440422 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:34:07.440391    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644\": container with ID starting with 68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644 not found: ID does not exist" containerID="68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644"
Nov 02 18:34:07.440527 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.440433    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644"} err="failed to get container status \"68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644\": rpc error: code = NotFound desc = could not find container \"68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644\": container with ID starting with 68581e8db017162ea113c7ca911f421804dc9b651613778372dd6378bfea4644 not found: ID does not exist"
Nov 02 18:34:07.588872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.588768    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"]
Nov 02 18:34:07.592404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.592376    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-69s4x"]
Nov 02 18:34:07.866054 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:07.865974    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" path="/var/lib/kubelet/pods/07985da9-31f1-4365-8a90-9be674b0e6ab/volumes"
Nov 02 18:34:12.774179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:12.774114    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" podUID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:34:13.468872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.468836    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"]
Nov 02 18:34:13.469495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469481    4956 cpu_manager.go:414] "containerMap: removing container" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy"
Nov 02 18:34:13.469562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469496    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy"
Nov 02 18:34:13.469562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469522    4956 cpu_manager.go:414] "containerMap: removing container" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing"
Nov 02 18:34:13.469562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469527    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing"
Nov 02 18:34:13.469562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469554    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" containerName="revoke-keys"
Nov 02 18:34:13.469562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469559    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" containerName="revoke-keys"
Nov 02 18:34:13.469720 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469649    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="07985da9-31f1-4365-8a90-9be674b0e6ab" containerName="payload-processing"
Nov 02 18:34:13.469720 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469663    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" containerName="revoke-keys"
Nov 02 18:34:13.469720 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.469671    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ed5b33-070f-49e9-a5c2-635c0b220827" containerName="istio-proxy"
Nov 02 18:34:13.475007 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.474971    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:13.528412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.528383    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"]
Nov 02 18:34:13.574230 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.574186    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-134555-dockercfg-7p69m\""
Nov 02 18:34:13.574230 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.574215    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-134555\""
Nov 02 18:34:13.608039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.608004    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gjg\" (UniqueName: \"kubernetes.io/projected/5945f88d-c0bb-46b3-a915-3f0f17b84217-kube-api-access-42gjg\") pod \"payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:13.608039 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.608044    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5945f88d-c0bb-46b3-a915-3f0f17b84217-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:13.710106 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.710066    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gjg\" (UniqueName: \"kubernetes.io/projected/5945f88d-c0bb-46b3-a915-3f0f17b84217-kube-api-access-42gjg\") pod \"payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:13.710324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.710141    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5945f88d-c0bb-46b3-a915-3f0f17b84217-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:13.710934 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.710914    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5945f88d-c0bb-46b3-a915-3f0f17b84217-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:13.766818 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.766746    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gjg\" (UniqueName: \"kubernetes.io/projected/5945f88d-c0bb-46b3-a915-3f0f17b84217-kube-api-access-42gjg\") pod \"payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:13.789716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:13.789684    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:14.163532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:14.163501    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"]
Nov 02 18:34:14.164347 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:34:14.164318    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5945f88d_c0bb_46b3_a915_3f0f17b84217.slice/crio-b29ca0e09170772d7ae6bb33b1521854062a70c1dcad6066029d1dd85c98ee13 WatchSource:0}: Error finding container b29ca0e09170772d7ae6bb33b1521854062a70c1dcad6066029d1dd85c98ee13: Status 404 returned error can't find the container with id b29ca0e09170772d7ae6bb33b1521854062a70c1dcad6066029d1dd85c98ee13
Nov 02 18:34:14.477546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:14.477510    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" event={"ID":"5945f88d-c0bb-46b3-a915-3f0f17b84217","Type":"ContainerStarted","Data":"4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449"}
Nov 02 18:34:14.477746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:14.477586    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" event={"ID":"5945f88d-c0bb-46b3-a915-3f0f17b84217","Type":"ContainerStarted","Data":"b29ca0e09170772d7ae6bb33b1521854062a70c1dcad6066029d1dd85c98ee13"}
Nov 02 18:34:14.477746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:14.477641    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:14.595071 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:14.595004    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" podStartSLOduration=1.594985371 podStartE2EDuration="1.594985371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:14.590897906 +0000 UTC m=+4284.847928526" watchObservedRunningTime="2026-11-02 18:34:14.594985371 +0000 UTC m=+4284.852015992"
Nov 02 18:34:22.775001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:22.774961    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" podUID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:34:23.618821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:23.618782    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"]
Nov 02 18:34:23.729882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:23.729850    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"]
Nov 02 18:34:25.486704 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:25.486670    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:34:27.296555 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.296494    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:34:27.384262 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.384226    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-plugins-config-volume\") pod \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") "
Nov 02 18:34:27.384459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.384373    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials\") pod \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") "
Nov 02 18:34:27.384459 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.384402    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-routing-overlay-volume\") pod \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") "
Nov 02 18:34:27.384595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.384580    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-kube-api-access-mspt4\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-kube-api-access-mspt4\") pod \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\" (UID: \"cd9a4a34-abee-461e-a01a-3f1f89ba22eb\") "
Nov 02 18:34:27.384753 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.384645    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-plugins-config-volume" pod "cd9a4a34-abee-461e-a01a-3f1f89ba22eb" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:27.384849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.384825    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-routing-overlay-volume" pod "cd9a4a34-abee-461e-a01a-3f1f89ba22eb" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:27.385055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.385042    4956 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-routing-overlay-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:27.385101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.385058    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:27.386629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.386605    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials" pod "cd9a4a34-abee-461e-a01a-3f1f89ba22eb" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:27.386718 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.386656    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-kube-api-access-mspt4" pod "cd9a4a34-abee-461e-a01a-3f1f89ba22eb" (UID: "cd9a4a34-abee-461e-a01a-3f1f89ba22eb"). InnerVolumeSpecName "kube-api-access-mspt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:27.486374 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.486339    4956 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-provider-credentials\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:27.486374 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.486365    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mspt4\" (UniqueName: \"kubernetes.io/projected/cd9a4a34-abee-461e-a01a-3f1f89ba22eb-kube-api-access-mspt4\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:27.566704 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.566616    4956 generic.go:356] "Generic (PLEG): container finished" podID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" containerID="e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464" exitCode=0
Nov 02 18:34:27.566704 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.566677    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"
Nov 02 18:34:27.566704 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.566682    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" event={"ID":"cd9a4a34-abee-461e-a01a-3f1f89ba22eb","Type":"ContainerDied","Data":"e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464"}
Nov 02 18:34:27.566957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.566741    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w" event={"ID":"cd9a4a34-abee-461e-a01a-3f1f89ba22eb","Type":"ContainerDied","Data":"8b203eef1b0c7b0b675bfb75bfc53df0ea9dddc5c7345175c1ac6af759a54dab"}
Nov 02 18:34:27.566957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.566770    4956 scope.go:122] "RemoveContainer" containerID="e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464"
Nov 02 18:34:27.579364 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.579293    4956 scope.go:122] "RemoveContainer" containerID="e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464"
Nov 02 18:34:27.579826 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:34:27.579802    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464\": container with ID starting with e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464 not found: ID does not exist" containerID="e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464"
Nov 02 18:34:27.579889 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.579839    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464"} err="failed to get container status \"e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464\": rpc error: code = NotFound desc = could not find container \"e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464\": container with ID starting with e36e85f2bf86039b08bb5296f36263911393d92cbdff8bca8c2520b5cc4af464 not found: ID does not exist"
Nov 02 18:34:27.732952 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.732914    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"]
Nov 02 18:34:27.736657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.736631    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8ddf7bb45-qzn7w"]
Nov 02 18:34:27.865722 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:27.865644    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" path="/var/lib/kubelet/pods/cd9a4a34-abee-461e-a01a-3f1f89ba22eb/volumes"
Nov 02 18:34:28.619865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:28.619816    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb" podUID="546e4fdc-ee01-438f-9c4e-3bc4a611536a" containerName="payload-processing" containerID="cri-o://4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5" gracePeriod=25
Nov 02 18:34:28.730882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:28.730837    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl" podUID="60844996-53ca-4873-a67e-14d51a48f8b6" containerName="payload-pre-processing" containerID="cri-o://f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd" gracePeriod=25
Nov 02 18:34:28.905832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:28.905804    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:34:29.005317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.005290    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546e4fdc-ee01-438f-9c4e-3bc4a611536a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546e4fdc-ee01-438f-9c4e-3bc4a611536a-plugins-config-volume\") pod \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") "
Nov 02 18:34:29.005460 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.005390    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546e4fdc-ee01-438f-9c4e-3bc4a611536a-kube-api-access-kcf7j\" (UniqueName: \"kubernetes.io/projected/546e4fdc-ee01-438f-9c4e-3bc4a611536a-kube-api-access-kcf7j\") pod \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\" (UID: \"546e4fdc-ee01-438f-9c4e-3bc4a611536a\") "
Nov 02 18:34:29.005710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.005685    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546e4fdc-ee01-438f-9c4e-3bc4a611536a-plugins-config-volume" pod "546e4fdc-ee01-438f-9c4e-3bc4a611536a" (UID: "546e4fdc-ee01-438f-9c4e-3bc4a611536a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:29.006013 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.005998    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546e4fdc-ee01-438f-9c4e-3bc4a611536a-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:29.007726 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.007699    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546e4fdc-ee01-438f-9c4e-3bc4a611536a-kube-api-access-kcf7j" pod "546e4fdc-ee01-438f-9c4e-3bc4a611536a" (UID: "546e4fdc-ee01-438f-9c4e-3bc4a611536a"). InnerVolumeSpecName "kube-api-access-kcf7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:29.016628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.016609    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:34:29.053686 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:34:29.053648    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:34:29.107164 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.107133    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60844996-53ca-4873-a67e-14d51a48f8b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60844996-53ca-4873-a67e-14d51a48f8b6-plugins-config-volume\") pod \"60844996-53ca-4873-a67e-14d51a48f8b6\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") "
Nov 02 18:34:29.107426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.107206    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60844996-53ca-4873-a67e-14d51a48f8b6-kube-api-access-n2kmb\" (UniqueName: \"kubernetes.io/projected/60844996-53ca-4873-a67e-14d51a48f8b6-kube-api-access-n2kmb\") pod \"60844996-53ca-4873-a67e-14d51a48f8b6\" (UID: \"60844996-53ca-4873-a67e-14d51a48f8b6\") "
Nov 02 18:34:29.107573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.107544    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60844996-53ca-4873-a67e-14d51a48f8b6-plugins-config-volume" pod "60844996-53ca-4873-a67e-14d51a48f8b6" (UID: "60844996-53ca-4873-a67e-14d51a48f8b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:29.107838 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.107818    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60844996-53ca-4873-a67e-14d51a48f8b6-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:29.108095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.107842    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcf7j\" (UniqueName: \"kubernetes.io/projected/546e4fdc-ee01-438f-9c4e-3bc4a611536a-kube-api-access-kcf7j\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:29.110002 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.109966    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60844996-53ca-4873-a67e-14d51a48f8b6-kube-api-access-n2kmb" pod "60844996-53ca-4873-a67e-14d51a48f8b6" (UID: "60844996-53ca-4873-a67e-14d51a48f8b6"). InnerVolumeSpecName "kube-api-access-n2kmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:29.209395 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.209363    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2kmb\" (UniqueName: \"kubernetes.io/projected/60844996-53ca-4873-a67e-14d51a48f8b6-kube-api-access-n2kmb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:29.581948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.581848    4956 generic.go:356] "Generic (PLEG): container finished" podID="546e4fdc-ee01-438f-9c4e-3bc4a611536a" containerID="4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5" exitCode=0
Nov 02 18:34:29.581948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.581908    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"
Nov 02 18:34:29.581948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.581932    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb" event={"ID":"546e4fdc-ee01-438f-9c4e-3bc4a611536a","Type":"ContainerDied","Data":"4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5"}
Nov 02 18:34:29.582280 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.582019    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb" event={"ID":"546e4fdc-ee01-438f-9c4e-3bc4a611536a","Type":"ContainerDied","Data":"a02e72c195fe3265c0b1f190a27b9ab478db2baf369fc383ba998a4cb522484d"}
Nov 02 18:34:29.582280 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.582051    4956 scope.go:122] "RemoveContainer" containerID="4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5"
Nov 02 18:34:29.583638 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.583606    4956 generic.go:356] "Generic (PLEG): container finished" podID="60844996-53ca-4873-a67e-14d51a48f8b6" containerID="f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd" exitCode=0
Nov 02 18:34:29.583924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.583664    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"
Nov 02 18:34:29.583924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.583704    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl" event={"ID":"60844996-53ca-4873-a67e-14d51a48f8b6","Type":"ContainerDied","Data":"f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd"}
Nov 02 18:34:29.583924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.583748    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl" event={"ID":"60844996-53ca-4873-a67e-14d51a48f8b6","Type":"ContainerDied","Data":"d0590883c752df685cde683ceff79f8e6bbe6a00850653b61389531133b232f3"}
Nov 02 18:34:29.594868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.594853    4956 scope.go:122] "RemoveContainer" containerID="4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5"
Nov 02 18:34:29.595211 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:34:29.595191    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5\": container with ID starting with 4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5 not found: ID does not exist" containerID="4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5"
Nov 02 18:34:29.595299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.595226    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5"} err="failed to get container status \"4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5\": rpc error: code = NotFound desc = could not find container \"4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5\": container with ID starting with 4ec9f0a7ee35cb77473801055f8c7181fabe9002dd1523ca088ddb5666bbd4c5 not found: ID does not exist"
Nov 02 18:34:29.595299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.595245    4956 scope.go:122] "RemoveContainer" containerID="f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd"
Nov 02 18:34:29.604506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.604489    4956 scope.go:122] "RemoveContainer" containerID="f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd"
Nov 02 18:34:29.604755 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:34:29.604735    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd\": container with ID starting with f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd not found: ID does not exist" containerID="f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd"
Nov 02 18:34:29.604806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.604762    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd"} err="failed to get container status \"f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd\": rpc error: code = NotFound desc = could not find container \"f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd\": container with ID starting with f3a8317fbd6dfcb43bf28b9cfe60b1ccced39d10de416d79c1e39c31f7e9fdfd not found: ID does not exist"
Nov 02 18:34:29.745000 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.744962    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"]
Nov 02 18:34:29.748112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.748081    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-740c43-5788d44479-hj2dl"]
Nov 02 18:34:29.856556 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.856454    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"]
Nov 02 18:34:29.868732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.868702    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60844996-53ca-4873-a67e-14d51a48f8b6" path="/var/lib/kubelet/pods/60844996-53ca-4873-a67e-14d51a48f8b6/volumes"
Nov 02 18:34:29.869016 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:29.869002    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-740c43-855ccdb667-qtgmb"]
Nov 02 18:34:31.866568 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:31.866534    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546e4fdc-ee01-438f-9c4e-3bc4a611536a" path="/var/lib/kubelet/pods/546e4fdc-ee01-438f-9c4e-3bc4a611536a/volumes"
Nov 02 18:34:44.332405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.332359    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"]
Nov 02 18:34:44.333279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333260    4956 cpu_manager.go:414] "containerMap: removing container" podUID="60844996-53ca-4873-a67e-14d51a48f8b6" containerName="payload-pre-processing"
Nov 02 18:34:44.333346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333281    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60844996-53ca-4873-a67e-14d51a48f8b6" containerName="payload-pre-processing"
Nov 02 18:34:44.333346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333304    4956 cpu_manager.go:414] "containerMap: removing container" podUID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" containerName="payload-processing"
Nov 02 18:34:44.333346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333311    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" containerName="payload-processing"
Nov 02 18:34:44.333470 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333371    4956 cpu_manager.go:414] "containerMap: removing container" podUID="546e4fdc-ee01-438f-9c4e-3bc4a611536a" containerName="payload-processing"
Nov 02 18:34:44.333470 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333379    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546e4fdc-ee01-438f-9c4e-3bc4a611536a" containerName="payload-processing"
Nov 02 18:34:44.333541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333515    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="60844996-53ca-4873-a67e-14d51a48f8b6" containerName="payload-pre-processing"
Nov 02 18:34:44.333541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333528    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9a4a34-abee-461e-a01a-3f1f89ba22eb" containerName="payload-processing"
Nov 02 18:34:44.333643 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.333547    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="546e4fdc-ee01-438f-9c4e-3bc4a611536a" containerName="payload-processing"
Nov 02 18:34:44.343031 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.343010    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.381086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.381051    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"]
Nov 02 18:34:44.385385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.385369    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.482017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.481987    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"]
Nov 02 18:34:44.492037 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492000    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.492275 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492063    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb274a7-029a-463d-98be-a5b61361ed63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.492275 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492186    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.492384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492270    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.492384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492320    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.492384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492371    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.492537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492516    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7s5r\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-kube-api-access-r7s5r\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.492596 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492561    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.492717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492625    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.492717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492662    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.492717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492691    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.492853 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492797    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.492853 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492843    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb274a7-029a-463d-98be-a5b61361ed63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.492949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492880    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.492949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.492936    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5nd\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-kube-api-access-dg5nd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.493082 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.493023    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.493186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.493166    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.493303 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.493283    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.539191 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.539156    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"]
Nov 02 18:34:44.594786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.594700    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.594786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.594753    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.594786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.594782    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb274a7-029a-463d-98be-a5b61361ed63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.594807    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.594827    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.594852    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595082    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595199    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s5r\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-kube-api-access-r7s5r\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.595291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595249    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.595468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595293    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595334    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595363    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595435    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.595707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595473    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.595707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595507    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.595707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595546    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595610    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.595707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595635    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb274a7-029a-463d-98be-a5b61361ed63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.595707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595641    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.596005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595729    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.596005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595757    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb274a7-029a-463d-98be-a5b61361ed63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.596005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595791    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.596005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595824    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5nd\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-kube-api-access-dg5nd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.596005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595843    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.596005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595900    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.596005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.595935    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.596374 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.596033    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.596374 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.596319    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.597450 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.597425    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.598245 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.598224    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.598764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.598748    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.598893 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.598872    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb274a7-029a-463d-98be-a5b61361ed63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.651480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.651444    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.651986 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.651961    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.652111 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.652015    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5nd\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-kube-api-access-dg5nd\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-txlv4\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.652111 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.652071    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s5r\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-kube-api-access-r7s5r\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqnqs\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:44.658003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.657982    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:44.698800 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:44.698772    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:45.090454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.090400    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"]
Nov 02 18:34:45.091189 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:34:45.090895    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb274a7_029a_463d_98be_a5b61361ed63.slice/crio-39bd015949359ec2c178181c00458097c95f114f17717bab047392a966d1f34a WatchSource:0}: Error finding container 39bd015949359ec2c178181c00458097c95f114f17717bab047392a966d1f34a: Status 404 returned error can't find the container with id 39bd015949359ec2c178181c00458097c95f114f17717bab047392a966d1f34a
Nov 02 18:34:45.093846 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.093807    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:34:45.093918 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.093894    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:34:45.093954 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.093932    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:34:45.094210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.094189    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"]
Nov 02 18:34:45.130250 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:34:45.130222    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cd6c76_392c_4f6d_a654_4a86fa888a47.slice/crio-22ae9ea736008b514c5410b2bdef885984039c2e3e32aa9c1c0abd81471a949f WatchSource:0}: Error finding container 22ae9ea736008b514c5410b2bdef885984039c2e3e32aa9c1c0abd81471a949f: Status 404 returned error can't find the container with id 22ae9ea736008b514c5410b2bdef885984039c2e3e32aa9c1c0abd81471a949f
Nov 02 18:34:45.132948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.132919    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:34:45.133017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.132994    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:34:45.133056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.133021    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:34:45.697386 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.697351    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" event={"ID":"f3cd6c76-392c-4f6d-a654-4a86fa888a47","Type":"ContainerStarted","Data":"dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2"}
Nov 02 18:34:45.697821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.697415    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" event={"ID":"f3cd6c76-392c-4f6d-a654-4a86fa888a47","Type":"ContainerStarted","Data":"22ae9ea736008b514c5410b2bdef885984039c2e3e32aa9c1c0abd81471a949f"}
Nov 02 18:34:45.698905 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.698880    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:45.699338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.699309    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" event={"ID":"fdb274a7-029a-463d-98be-a5b61361ed63","Type":"ContainerStarted","Data":"d866a2b44f36b5077f54254e0b2a0a54be0a70bc713606acce22fa67550b33ea"}
Nov 02 18:34:45.699472 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.699351    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" event={"ID":"fdb274a7-029a-463d-98be-a5b61361ed63","Type":"ContainerStarted","Data":"39bd015949359ec2c178181c00458097c95f114f17717bab047392a966d1f34a"}
Nov 02 18:34:45.816777 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.816723    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podStartSLOduration=1.81670869 podStartE2EDuration="1.81670869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:45.814441239 +0000 UTC m=+4316.071471859" watchObservedRunningTime="2026-11-02 18:34:45.81670869 +0000 UTC m=+4316.073739376"
Nov 02 18:34:45.978174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:45.967570    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podStartSLOduration=1.967548063 podStartE2EDuration="1.967548063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:45.945690821 +0000 UTC m=+4316.202721440" watchObservedRunningTime="2026-11-02 18:34:45.967548063 +0000 UTC m=+4316.224578683"
Nov 02 18:34:46.381963 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.381873    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"]
Nov 02 18:34:46.386190 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.386167    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.423727 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.423695    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89r4h\" (UniqueName: \"kubernetes.io/projected/3e21fa92-9e3a-40dd-b8b4-029075b53017-kube-api-access-89r4h\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.423878 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.423784    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.423878 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.423858    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.448146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.427701    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"]
Nov 02 18:34:46.448146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.438517    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:34:46.448146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.438626    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:34:46.549144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.525397    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.549144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.525497    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.549144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.525697    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89r4h\" (UniqueName: \"kubernetes.io/projected/3e21fa92-9e3a-40dd-b8b4-029075b53017-kube-api-access-89r4h\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.549144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.526809    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.579167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.578958    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.581970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.581942    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89r4h\" (UniqueName: \"kubernetes.io/projected/3e21fa92-9e3a-40dd-b8b4-029075b53017-kube-api-access-89r4h\") pod \"maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.673207 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.672504    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:46.678145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.677955    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:34:46.678145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.678014    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 18:34:46.701043 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.701018    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:46.701444 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.701034    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:34:46.701444 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:46.701097    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 18:34:47.184165 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:47.183496    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"]
Nov 02 18:34:47.662268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:47.662224    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:34:47.662667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:47.662636    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 18:34:47.700045 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:47.700014    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:34:47.700312 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:47.700288    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 18:34:47.714041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:47.714011    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j" event={"ID":"3e21fa92-9e3a-40dd-b8b4-029075b53017","Type":"ContainerStarted","Data":"b4a00c2b818700f78c5e994e1b0f0180e6c6a1afd95e58c98274dd15770b86f2"}
Nov 02 18:34:48.658799 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:48.658757    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:34:48.659175 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:48.659144    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 18:34:48.699929 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:48.699899    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:34:48.700188 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:48.700164    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 18:34:48.722304 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:48.722266    4956 generic.go:356] "Generic (PLEG): container finished" podID="3e21fa92-9e3a-40dd-b8b4-029075b53017" containerID="fbae76810d89483ac9f8af18f4e090dfec239f53effed634027084d3dafc8979" exitCode=0
Nov 02 18:34:48.722931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:48.722581    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j" event={"ID":"3e21fa92-9e3a-40dd-b8b4-029075b53017","Type":"ContainerDied","Data":"fbae76810d89483ac9f8af18f4e090dfec239f53effed634027084d3dafc8979"}
Nov 02 18:34:49.659573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:49.659518    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:34:49.659897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:49.659866    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 18:34:49.700155 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:49.700104    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:34:49.700395 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:49.700375    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 18:34:50.085615 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.085590    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:50.113571 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.113534    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-openshift-ca\") pod \"3e21fa92-9e3a-40dd-b8b4-029075b53017\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") "
Nov 02 18:34:50.113731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.113674    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e21fa92-9e3a-40dd-b8b4-029075b53017-kube-api-access-89r4h\" (UniqueName: \"kubernetes.io/projected/3e21fa92-9e3a-40dd-b8b4-029075b53017-kube-api-access-89r4h\") pod \"3e21fa92-9e3a-40dd-b8b4-029075b53017\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") "
Nov 02 18:34:50.113731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.113707    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-xks-ca\") pod \"3e21fa92-9e3a-40dd-b8b4-029075b53017\" (UID: \"3e21fa92-9e3a-40dd-b8b4-029075b53017\") "
Nov 02 18:34:50.113900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.113874    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-openshift-ca" pod "3e21fa92-9e3a-40dd-b8b4-029075b53017" (UID: "3e21fa92-9e3a-40dd-b8b4-029075b53017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:50.114028 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.114014    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-openshift-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:50.115900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.115873    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e21fa92-9e3a-40dd-b8b4-029075b53017-kube-api-access-89r4h" pod "3e21fa92-9e3a-40dd-b8b4-029075b53017" (UID: "3e21fa92-9e3a-40dd-b8b4-029075b53017"). InnerVolumeSpecName "kube-api-access-89r4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:50.116019 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.115952    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-xks-ca" pod "3e21fa92-9e3a-40dd-b8b4-029075b53017" (UID: "3e21fa92-9e3a-40dd-b8b4-029075b53017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:50.215025 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.214984    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89r4h\" (UniqueName: \"kubernetes.io/projected/3e21fa92-9e3a-40dd-b8b4-029075b53017-kube-api-access-89r4h\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:50.215025 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.215020    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e21fa92-9e3a-40dd-b8b4-029075b53017-maas-api-xks-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:34:50.659146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.658561    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:34:50.659146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.658630    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 18:34:50.700194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.700153    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:34:50.700370 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.700215    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 18:34:50.783189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.782287    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j" event={"ID":"3e21fa92-9e3a-40dd-b8b4-029075b53017","Type":"ContainerDied","Data":"b4a00c2b818700f78c5e994e1b0f0180e6c6a1afd95e58c98274dd15770b86f2"}
Nov 02 18:34:50.783189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.782337    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a00c2b818700f78c5e994e1b0f0180e6c6a1afd95e58c98274dd15770b86f2"
Nov 02 18:34:50.783189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:50.782401    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"
Nov 02 18:34:51.659211 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:51.659169    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:34:51.659703 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:51.659235    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 18:34:51.699737 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:51.699687    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:34:51.699961 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:51.699745    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 18:34:52.658986 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:52.658926    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:34:52.659215 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:52.659006    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 18:34:52.699216 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:52.699164    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:34:52.699753 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:52.699238    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 18:34:53.663793 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:53.663762    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:53.664251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:53.664230    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:53.665091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:53.665075    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:34:53.704205 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:53.704177    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:53.704577 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:53.704493    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:53.705214 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:34:53.705195    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:34:59.053751 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:34:59.053711    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:35:12.456202 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.456159    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"]
Nov 02 18:35:12.456888 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.456871    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3e21fa92-9e3a-40dd-b8b4-029075b53017" containerName="revoke-keys"
Nov 02 18:35:12.456951 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.456890    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e21fa92-9e3a-40dd-b8b4-029075b53017" containerName="revoke-keys"
Nov 02 18:35:12.456995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.456985    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e21fa92-9e3a-40dd-b8b4-029075b53017" containerName="revoke-keys"
Nov 02 18:35:12.460692 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.460676    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.507804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.507763    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.508000 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.507889    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzxtm\" (UniqueName: \"kubernetes.io/projected/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-kube-api-access-nzxtm\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.508047 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.508022    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.516481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.516453    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"]
Nov 02 18:35:12.561347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.561310    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:35:12.561553 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.561382    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:35:12.609172 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.609098    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.609374 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.609232    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzxtm\" (UniqueName: \"kubernetes.io/projected/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-kube-api-access-nzxtm\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.609449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.609368    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.609919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.609891    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.612090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.612061    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.665406 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.665365    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzxtm\" (UniqueName: \"kubernetes.io/projected/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-kube-api-access-nzxtm\") pod \"maas-api-revoke-keys-e2e-inf-a-546881-jfhx9\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:12.774862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:12.774777    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:13.152223 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:13.152190    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"]
Nov 02 18:35:13.153345 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:35:13.153323    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1779fcf3_5381_4b3a_ae53_f689b5ce6cac.slice/crio-6bf9e822b545a985e052ac44288716895e65f62f76ce9b935518a9d733eed99b WatchSource:0}: Error finding container 6bf9e822b545a985e052ac44288716895e65f62f76ce9b935518a9d733eed99b: Status 404 returned error can't find the container with id 6bf9e822b545a985e052ac44288716895e65f62f76ce9b935518a9d733eed99b
Nov 02 18:35:13.939910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:13.939839    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9" event={"ID":"1779fcf3-5381-4b3a-ae53-f689b5ce6cac","Type":"ContainerStarted","Data":"d28bf54faca708ee7745c451683cdc85604a75256f9e39c0d8d7671a49665463"}
Nov 02 18:35:13.940568 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:13.939954    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9" event={"ID":"1779fcf3-5381-4b3a-ae53-f689b5ce6cac","Type":"ContainerStarted","Data":"6bf9e822b545a985e052ac44288716895e65f62f76ce9b935518a9d733eed99b"}
Nov 02 18:35:14.054620 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:14.054564    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9" podStartSLOduration=2.054550472 podStartE2EDuration="2.054550472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:35:14.051408149 +0000 UTC m=+4344.308438768" watchObservedRunningTime="2026-11-02 18:35:14.054550472 +0000 UTC m=+4344.311581091"
Nov 02 18:35:14.947451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:14.947411    4956 generic.go:356] "Generic (PLEG): container finished" podID="1779fcf3-5381-4b3a-ae53-f689b5ce6cac" containerID="d28bf54faca708ee7745c451683cdc85604a75256f9e39c0d8d7671a49665463" exitCode=0
Nov 02 18:35:14.947883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:14.947497    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9" event={"ID":"1779fcf3-5381-4b3a-ae53-f689b5ce6cac","Type":"ContainerDied","Data":"d28bf54faca708ee7745c451683cdc85604a75256f9e39c0d8d7671a49665463"}
Nov 02 18:35:16.112539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.112508    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:16.152743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.152703    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-kube-api-access-nzxtm\" (UniqueName: \"kubernetes.io/projected/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-kube-api-access-nzxtm\") pod \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") "
Nov 02 18:35:16.152968 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.152949    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-openshift-ca\") pod \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") "
Nov 02 18:35:16.153088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.153073    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-xks-ca\") pod \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\" (UID: \"1779fcf3-5381-4b3a-ae53-f689b5ce6cac\") "
Nov 02 18:35:16.153372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.153342    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-openshift-ca" pod "1779fcf3-5381-4b3a-ae53-f689b5ce6cac" (UID: "1779fcf3-5381-4b3a-ae53-f689b5ce6cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:35:16.153524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.153506    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-openshift-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:16.155091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.155065    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-kube-api-access-nzxtm" pod "1779fcf3-5381-4b3a-ae53-f689b5ce6cac" (UID: "1779fcf3-5381-4b3a-ae53-f689b5ce6cac"). InnerVolumeSpecName "kube-api-access-nzxtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:16.155204 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.155131    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-xks-ca" pod "1779fcf3-5381-4b3a-ae53-f689b5ce6cac" (UID: "1779fcf3-5381-4b3a-ae53-f689b5ce6cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:35:16.254656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.254572    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-maas-api-xks-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:16.254656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.254603    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzxtm\" (UniqueName: \"kubernetes.io/projected/1779fcf3-5381-4b3a-ae53-f689b5ce6cac-kube-api-access-nzxtm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:16.962583 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.962550    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"
Nov 02 18:35:16.962765 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.962581    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9" event={"ID":"1779fcf3-5381-4b3a-ae53-f689b5ce6cac","Type":"ContainerDied","Data":"6bf9e822b545a985e052ac44288716895e65f62f76ce9b935518a9d733eed99b"}
Nov 02 18:35:16.962765 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:16.962628    4956 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bf9e822b545a985e052ac44288716895e65f62f76ce9b935518a9d733eed99b"
Nov 02 18:35:29.056099 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:35:29.056062    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:35:34.833286 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:34.833199    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"]
Nov 02 18:35:34.881969 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:34.881936    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70712-fsq65"]
Nov 02 18:35:35.865488 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:35.865455    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db4aee1-079a-44f5-b43f-2d5ef6a56fb9" path="/var/lib/kubelet/pods/1db4aee1-079a-44f5-b43f-2d5ef6a56fb9/volumes"
Nov 02 18:35:39.361622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.361584    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"]
Nov 02 18:35:39.362340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.362324    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1779fcf3-5381-4b3a-ae53-f689b5ce6cac" containerName="revoke-keys"
Nov 02 18:35:39.362401 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.362341    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1779fcf3-5381-4b3a-ae53-f689b5ce6cac" containerName="revoke-keys"
Nov 02 18:35:39.362467 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.362454    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1779fcf3-5381-4b3a-ae53-f689b5ce6cac" containerName="revoke-keys"
Nov 02 18:35:39.366210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.366192    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.418712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.418664    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-134555-serving-cert\""
Nov 02 18:35:39.422156 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.422106    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"]
Nov 02 18:35:39.464904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.464864    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.465081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.464932    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gphr\" (UniqueName: \"kubernetes.io/projected/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-kube-api-access-9gphr\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.465081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.465041    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.465081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.465071    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-metrics-certs\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.566355 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.566314    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.566586 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.566374    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gphr\" (UniqueName: \"kubernetes.io/projected/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-kube-api-access-9gphr\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.566586 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.566418    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.566586 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.566433    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-metrics-certs\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.566586 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:35:39.566489    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-serving-cert: secret "maas-api-e2e-inf-a-134555-serving-cert" not found
Nov 02 18:35:39.566586 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:35:39.566580    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls podName:5c125e9c-b65c-40bd-8313-c3f6c6b87de1 nodeName:}" failed. No retries permitted until 2026-11-02 18:35:40.066556447 +0000 UTC m=+4370.323587066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls") pod "maas-api-e2e-inf-a-134555-79567cb548-87sz7" (UID: "5c125e9c-b65c-40bd-8313-c3f6c6b87de1") : secret "maas-api-e2e-inf-a-134555-serving-cert" not found
Nov 02 18:35:39.567219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.567196    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.569352 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.569330    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-metrics-certs\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:39.624159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:39.624066    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gphr\" (UniqueName: \"kubernetes.io/projected/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-kube-api-access-9gphr\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:40.073669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:40.073626    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:40.076299 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:40.076268    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls\") pod \"maas-api-e2e-inf-a-134555-79567cb548-87sz7\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:40.280013 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:40.279974    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:40.658685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:40.658652    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"]
Nov 02 18:35:40.659808 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:35:40.659779    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c125e9c_b65c_40bd_8313_c3f6c6b87de1.slice/crio-1e049d7d88e02c0a2d6882f4ac717fea0c5d2e2df68feb3ab0fc4afc9b485dbb WatchSource:0}: Error finding container 1e049d7d88e02c0a2d6882f4ac717fea0c5d2e2df68feb3ab0fc4afc9b485dbb: Status 404 returned error can't find the container with id 1e049d7d88e02c0a2d6882f4ac717fea0c5d2e2df68feb3ab0fc4afc9b485dbb
Nov 02 18:35:40.661222 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:40.661199    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:35:41.125181 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:41.125112    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7" event={"ID":"5c125e9c-b65c-40bd-8313-c3f6c6b87de1","Type":"ContainerStarted","Data":"1e049d7d88e02c0a2d6882f4ac717fea0c5d2e2df68feb3ab0fc4afc9b485dbb"}
Nov 02 18:35:42.134296 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:42.134253    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7" event={"ID":"5c125e9c-b65c-40bd-8313-c3f6c6b87de1","Type":"ContainerStarted","Data":"ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b"}
Nov 02 18:35:42.134739 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:42.134345    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:42.249786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:42.249727    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7" podStartSLOduration=2.364931065 podStartE2EDuration="3.24971093s" totalImagesPullingTime="884.779865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:35:42.247609731 +0000 UTC m=+4372.504640351" watchObservedRunningTime="2026-11-02 18:35:42.24971093 +0000 UTC m=+4372.506741551"
Nov 02 18:35:48.146635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:35:48.146603    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:35:59.053108 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:35:59.053068    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:36:19.541370 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:19.541338    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"]
Nov 02 18:36:19.590144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:19.590097    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-662a80-fxv5j"]
Nov 02 18:36:19.867657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:19.867575    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e21fa92-9e3a-40dd-b8b4-029075b53017" path="/var/lib/kubelet/pods/3e21fa92-9e3a-40dd-b8b4-029075b53017/volumes"
Nov 02 18:36:27.907594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:27.907558    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"]
Nov 02 18:36:29.064364 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:36:29.064326    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:36:32.908725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:32.908678    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp" podUID="69e6b835-5238-4380-a558-c23cdeb8fbe3" containerName="payload-processing" containerID="cri-o://2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724" gracePeriod=25
Nov 02 18:36:33.177511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.177440    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:36:33.316690 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.316653    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e6b835-5238-4380-a558-c23cdeb8fbe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6b835-5238-4380-a558-c23cdeb8fbe3-plugins-config-volume\") pod \"69e6b835-5238-4380-a558-c23cdeb8fbe3\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") "
Nov 02 18:36:33.316987 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.316704    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e6b835-5238-4380-a558-c23cdeb8fbe3-kube-api-access-kgwv8\" (UniqueName: \"kubernetes.io/projected/69e6b835-5238-4380-a558-c23cdeb8fbe3-kube-api-access-kgwv8\") pod \"69e6b835-5238-4380-a558-c23cdeb8fbe3\" (UID: \"69e6b835-5238-4380-a558-c23cdeb8fbe3\") "
Nov 02 18:36:33.317182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.317133    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e6b835-5238-4380-a558-c23cdeb8fbe3-plugins-config-volume" pod "69e6b835-5238-4380-a558-c23cdeb8fbe3" (UID: "69e6b835-5238-4380-a558-c23cdeb8fbe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:33.317413 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.317386    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6b835-5238-4380-a558-c23cdeb8fbe3-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:33.319179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.319151    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e6b835-5238-4380-a558-c23cdeb8fbe3-kube-api-access-kgwv8" pod "69e6b835-5238-4380-a558-c23cdeb8fbe3" (UID: "69e6b835-5238-4380-a558-c23cdeb8fbe3"). InnerVolumeSpecName "kube-api-access-kgwv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:33.418636 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.418600    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgwv8\" (UniqueName: \"kubernetes.io/projected/69e6b835-5238-4380-a558-c23cdeb8fbe3-kube-api-access-kgwv8\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:33.538489 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.538448    4956 generic.go:356] "Generic (PLEG): container finished" podID="69e6b835-5238-4380-a558-c23cdeb8fbe3" containerID="2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724" exitCode=0
Nov 02 18:36:33.538702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.538493    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"
Nov 02 18:36:33.538702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.538535    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp" event={"ID":"69e6b835-5238-4380-a558-c23cdeb8fbe3","Type":"ContainerDied","Data":"2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724"}
Nov 02 18:36:33.538702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.538593    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp" event={"ID":"69e6b835-5238-4380-a558-c23cdeb8fbe3","Type":"ContainerDied","Data":"2f1d053d566066a68c78acd83e2a494ab1ee6a70e4f7bd547c3c12b307e6ebc8"}
Nov 02 18:36:33.538702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.538621    4956 scope.go:122] "RemoveContainer" containerID="2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724"
Nov 02 18:36:33.551988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.551969    4956 scope.go:122] "RemoveContainer" containerID="2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724"
Nov 02 18:36:33.552643 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:36:33.552622    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724\": container with ID starting with 2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724 not found: ID does not exist" containerID="2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724"
Nov 02 18:36:33.552747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.552651    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724"} err="failed to get container status \"2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724\": rpc error: code = NotFound desc = could not find container \"2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724\": container with ID starting with 2168330f4bcaafdbbfe8ccbda6efd2c224a37d98a5191a9c148f0a378b3e2724 not found: ID does not exist"
Nov 02 18:36:33.702259 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.702218    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"]
Nov 02 18:36:33.706532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.706509    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cf852-5b9849bd67-sg5lp"]
Nov 02 18:36:33.865655 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:33.865561    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e6b835-5238-4380-a558-c23cdeb8fbe3" path="/var/lib/kubelet/pods/69e6b835-5238-4380-a558-c23cdeb8fbe3/volumes"
Nov 02 18:36:39.418408 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.418360    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"]
Nov 02 18:36:39.418962 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.418609    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw" podUID="780d4861-313a-4b69-bf27-ce324a57639a" containerName="maas-api" containerID="cri-o://9ebe694ae7ea1bc599a82b56780d62db0d559a42fafccc06915b5b9c4c9baed2" gracePeriod=30
Nov 02 18:36:39.580718 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.580686    4956 generic.go:356] "Generic (PLEG): container finished" podID="780d4861-313a-4b69-bf27-ce324a57639a" containerID="9ebe694ae7ea1bc599a82b56780d62db0d559a42fafccc06915b5b9c4c9baed2" exitCode=0
Nov 02 18:36:39.580901 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.580770    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw" event={"ID":"780d4861-313a-4b69-bf27-ce324a57639a","Type":"ContainerDied","Data":"9ebe694ae7ea1bc599a82b56780d62db0d559a42fafccc06915b5b9c4c9baed2"}
Nov 02 18:36:39.710332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.710302    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:36:39.802455 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.802420    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-metrics-certs\") pod \"780d4861-313a-4b69-bf27-ce324a57639a\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") "
Nov 02 18:36:39.802667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.802483    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780d4861-313a-4b69-bf27-ce324a57639a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780d4861-313a-4b69-bf27-ce324a57639a-openshift-service-ca\") pod \"780d4861-313a-4b69-bf27-ce324a57639a\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") "
Nov 02 18:36:39.802667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.802577    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls\") pod \"780d4861-313a-4b69-bf27-ce324a57639a\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") "
Nov 02 18:36:39.802667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.802618    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780d4861-313a-4b69-bf27-ce324a57639a-kube-api-access-j2w29\" (UniqueName: \"kubernetes.io/projected/780d4861-313a-4b69-bf27-ce324a57639a-kube-api-access-j2w29\") pod \"780d4861-313a-4b69-bf27-ce324a57639a\" (UID: \"780d4861-313a-4b69-bf27-ce324a57639a\") "
Nov 02 18:36:39.802924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.802839    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780d4861-313a-4b69-bf27-ce324a57639a-openshift-service-ca" pod "780d4861-313a-4b69-bf27-ce324a57639a" (UID: "780d4861-313a-4b69-bf27-ce324a57639a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:39.803191 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.803170    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780d4861-313a-4b69-bf27-ce324a57639a-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:39.804915 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.804877    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls" pod "780d4861-313a-4b69-bf27-ce324a57639a" (UID: "780d4861-313a-4b69-bf27-ce324a57639a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:36:39.805504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.805478    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-metrics-certs" pod "780d4861-313a-4b69-bf27-ce324a57639a" (UID: "780d4861-313a-4b69-bf27-ce324a57639a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:36:39.805593 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.805530    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780d4861-313a-4b69-bf27-ce324a57639a-kube-api-access-j2w29" pod "780d4861-313a-4b69-bf27-ce324a57639a" (UID: "780d4861-313a-4b69-bf27-ce324a57639a"). InnerVolumeSpecName "kube-api-access-j2w29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:39.904832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.904797    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:39.904832 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.904831    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780d4861-313a-4b69-bf27-ce324a57639a-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:39.905035 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:39.904842    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2w29\" (UniqueName: \"kubernetes.io/projected/780d4861-313a-4b69-bf27-ce324a57639a-kube-api-access-j2w29\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:40.588427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:40.588396    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw" event={"ID":"780d4861-313a-4b69-bf27-ce324a57639a","Type":"ContainerDied","Data":"f8110b866e5757ce138fcc54c5e7a3cb9c24de537f48d5673ce74027977a98e2"}
Nov 02 18:36:40.588427 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:40.588425    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"
Nov 02 18:36:40.588940 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:40.588466    4956 scope.go:122] "RemoveContainer" containerID="9ebe694ae7ea1bc599a82b56780d62db0d559a42fafccc06915b5b9c4c9baed2"
Nov 02 18:36:40.752900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:40.752846    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"]
Nov 02 18:36:40.802428 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:40.802392    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-546881-58b7946789-6zmdw"]
Nov 02 18:36:41.865526 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:41.865492    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780d4861-313a-4b69-bf27-ce324a57639a" path="/var/lib/kubelet/pods/780d4861-313a-4b69-bf27-ce324a57639a/volumes"
Nov 02 18:36:41.877931 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:41.877903    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"]
Nov 02 18:36:41.926809 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:41.926779    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-546881-jfhx9"]
Nov 02 18:36:43.866562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:36:43.866520    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1779fcf3-5381-4b3a-ae53-f689b5ce6cac" path="/var/lib/kubelet/pods/1779fcf3-5381-4b3a-ae53-f689b5ce6cac/volumes"
Nov 02 18:36:59.053024 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:36:59.052974    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:37:25.375304 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:25.375263    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"]
Nov 02 18:37:25.485076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:25.485026    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"]
Nov 02 18:37:29.079286 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:37:29.079227    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:37:30.376096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.376040    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t" podUID="21e55d78-893c-4f4b-af11-41b397777898" containerName="payload-processing" containerID="cri-o://7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188" gracePeriod=25
Nov 02 18:37:30.486014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.485927    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt" podUID="5530c74a-b5e4-46fe-a7cc-0907e34b1afc" containerName="payload-pre-processing" containerID="cri-o://9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68" gracePeriod=25
Nov 02 18:37:30.655889 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.655855    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:37:30.719681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.719648    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e55d78-893c-4f4b-af11-41b397777898-kube-api-access-rdj7x\" (UniqueName: \"kubernetes.io/projected/21e55d78-893c-4f4b-af11-41b397777898-kube-api-access-rdj7x\") pod \"21e55d78-893c-4f4b-af11-41b397777898\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") "
Nov 02 18:37:30.719880 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.719850    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e55d78-893c-4f4b-af11-41b397777898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e55d78-893c-4f4b-af11-41b397777898-plugins-config-volume\") pod \"21e55d78-893c-4f4b-af11-41b397777898\" (UID: \"21e55d78-893c-4f4b-af11-41b397777898\") "
Nov 02 18:37:30.720332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.720304    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e55d78-893c-4f4b-af11-41b397777898-plugins-config-volume" pod "21e55d78-893c-4f4b-af11-41b397777898" (UID: "21e55d78-893c-4f4b-af11-41b397777898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:37:30.722130 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.722086    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e55d78-893c-4f4b-af11-41b397777898-kube-api-access-rdj7x" pod "21e55d78-893c-4f4b-af11-41b397777898" (UID: "21e55d78-893c-4f4b-af11-41b397777898"). InnerVolumeSpecName "kube-api-access-rdj7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:30.756851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.756825    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:37:30.820922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.820886    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-plugins-config-volume\") pod \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") "
Nov 02 18:37:30.821141 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.820936    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-kube-api-access-rscz6\" (UniqueName: \"kubernetes.io/projected/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-kube-api-access-rscz6\") pod \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\" (UID: \"5530c74a-b5e4-46fe-a7cc-0907e34b1afc\") "
Nov 02 18:37:30.821424 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.821393    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-plugins-config-volume" pod "5530c74a-b5e4-46fe-a7cc-0907e34b1afc" (UID: "5530c74a-b5e4-46fe-a7cc-0907e34b1afc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:37:30.821567 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.821545    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e55d78-893c-4f4b-af11-41b397777898-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:30.821642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.821574    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdj7x\" (UniqueName: \"kubernetes.io/projected/21e55d78-893c-4f4b-af11-41b397777898-kube-api-access-rdj7x\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:30.821642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.821589    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:30.823346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.823315    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-kube-api-access-rscz6" pod "5530c74a-b5e4-46fe-a7cc-0907e34b1afc" (UID: "5530c74a-b5e4-46fe-a7cc-0907e34b1afc"). InnerVolumeSpecName "kube-api-access-rscz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:30.923168 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.923043    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rscz6\" (UniqueName: \"kubernetes.io/projected/5530c74a-b5e4-46fe-a7cc-0907e34b1afc-kube-api-access-rscz6\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:30.973566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.973527    4956 generic.go:356] "Generic (PLEG): container finished" podID="5530c74a-b5e4-46fe-a7cc-0907e34b1afc" containerID="9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68" exitCode=0
Nov 02 18:37:30.973775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.973579    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"
Nov 02 18:37:30.973775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.973617    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt" event={"ID":"5530c74a-b5e4-46fe-a7cc-0907e34b1afc","Type":"ContainerDied","Data":"9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68"}
Nov 02 18:37:30.973775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.973678    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt" event={"ID":"5530c74a-b5e4-46fe-a7cc-0907e34b1afc","Type":"ContainerDied","Data":"54cd6a9e424f47ba28de74a515bb059fdaabcf75cbd943e644f5e848b9e291a8"}
Nov 02 18:37:30.973775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.973706    4956 scope.go:122] "RemoveContainer" containerID="9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68"
Nov 02 18:37:30.975187 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.975165    4956 generic.go:356] "Generic (PLEG): container finished" podID="21e55d78-893c-4f4b-af11-41b397777898" containerID="7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188" exitCode=0
Nov 02 18:37:30.975310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.975207    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"
Nov 02 18:37:30.975310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.975217    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t" event={"ID":"21e55d78-893c-4f4b-af11-41b397777898","Type":"ContainerDied","Data":"7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188"}
Nov 02 18:37:30.975310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.975253    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t" event={"ID":"21e55d78-893c-4f4b-af11-41b397777898","Type":"ContainerDied","Data":"d6e4ff6fccc66d4f8b2cfb4ac0b1cd9741be2d2bbe2fed5b69a2f182b6600071"}
Nov 02 18:37:30.984954 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.984934    4956 scope.go:122] "RemoveContainer" containerID="9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68"
Nov 02 18:37:30.985240 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:37:30.985221    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68\": container with ID starting with 9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68 not found: ID does not exist" containerID="9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68"
Nov 02 18:37:30.985335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.985250    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68"} err="failed to get container status \"9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68\": rpc error: code = NotFound desc = could not find container \"9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68\": container with ID starting with 9e97d037294d7657def8ec0630980741a4738704a0eb0b93de8fae5a20511f68 not found: ID does not exist"
Nov 02 18:37:30.985335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.985273    4956 scope.go:122] "RemoveContainer" containerID="7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188"
Nov 02 18:37:30.995277 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.995259    4956 scope.go:122] "RemoveContainer" containerID="7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188"
Nov 02 18:37:30.995577 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:37:30.995558    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188\": container with ID starting with 7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188 not found: ID does not exist" containerID="7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188"
Nov 02 18:37:30.995631 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:30.995585    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188"} err="failed to get container status \"7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188\": rpc error: code = NotFound desc = could not find container \"7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188\": container with ID starting with 7b54bfdbb0be4f54cecf4bba57d8af804d328fa03b9a50dc7121898a5037b188 not found: ID does not exist"
Nov 02 18:37:31.138402 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:31.138358    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"]
Nov 02 18:37:31.143333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:31.143306    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-608907-566c4f66d7-k8c7t"]
Nov 02 18:37:31.250770 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:31.250727    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"]
Nov 02 18:37:31.254831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:31.254795    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-608907-54c9dcf647-fvxzt"]
Nov 02 18:37:31.866033 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:31.866000    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e55d78-893c-4f4b-af11-41b397777898" path="/var/lib/kubelet/pods/21e55d78-893c-4f4b-af11-41b397777898/volumes"
Nov 02 18:37:31.866475 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:31.866458    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5530c74a-b5e4-46fe-a7cc-0907e34b1afc" path="/var/lib/kubelet/pods/5530c74a-b5e4-46fe-a7cc-0907e34b1afc/volumes"
Nov 02 18:37:50.527963 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:50.527933    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:37:50.534168 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:37:50.534106    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:37:59.052504 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:37:59.052464    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:38:29.052571 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:38:29.052534    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:38:52.018433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:52.018401    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"]
Nov 02 18:38:52.018984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:52.018616    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy" containerID="cri-o://5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9" gracePeriod=30
Nov 02 18:38:54.044608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:54.044566    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"]
Nov 02 18:38:54.047317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:54.047294    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67abda02-b4lp8"]
Nov 02 18:38:55.865751 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:55.865720    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f81d94f-0c20-436a-8df7-3aa4e6f121ab" path="/var/lib/kubelet/pods/3f81d94f-0c20-436a-8df7-3aa4e6f121ab/volumes"
Nov 02 18:38:57.306754 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.306726    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:38:57.397823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.397788    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-credential-socket\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.397823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.397824    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-certs\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.397869    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-kube-api-access-hnb9m\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-kube-api-access-hnb9m\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.397972    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-envoy\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398022    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee971e43-ac12-4c34-892f-a02c57da26e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee971e43-ac12-4c34-892f-a02c57da26e9-istiod-ca-cert\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398058    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-credential-socket" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:38:57.398217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398099    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-socket\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398198    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-podinfo\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398197    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-certs" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:38:57.398451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398275    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-data\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398326    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-token\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-token\") pod \"ee971e43-ac12-4c34-892f-a02c57da26e9\" (UID: \"ee971e43-ac12-4c34-892f-a02c57da26e9\") "
Nov 02 18:38:57.398451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398415    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-socket" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:38:57.398758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398466    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee971e43-ac12-4c34-892f-a02c57da26e9-istiod-ca-cert" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:38:57.398758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398545    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-data" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:38:57.398944 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398918    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee971e43-ac12-4c34-892f-a02c57da26e9-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.399138 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.398944    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.399138 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.399066    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.399332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.399140    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.399332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.399173    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.400727 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.400693    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-podinfo" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:38:57.400849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.400763    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-token" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:38:57.401092 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.401065    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-kube-api-access-hnb9m" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "kube-api-access-hnb9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:38:57.401481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.401457    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-envoy" pod "ee971e43-ac12-4c34-892f-a02c57da26e9" (UID: "ee971e43-ac12-4c34-892f-a02c57da26e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:38:57.501036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.500988    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.501036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.501038    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.501281 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.501054    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.501281 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.501068    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnb9m\" (UniqueName: \"kubernetes.io/projected/ee971e43-ac12-4c34-892f-a02c57da26e9-kube-api-access-hnb9m\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:57.619160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.619035    4956 generic.go:356] "Generic (PLEG): container finished" podID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerID="5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9" exitCode=0
Nov 02 18:38:57.619328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.619210    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" event={"ID":"ee971e43-ac12-4c34-892f-a02c57da26e9","Type":"ContainerDied","Data":"5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9"}
Nov 02 18:38:57.619328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.619232    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"
Nov 02 18:38:57.619328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.619299    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4" event={"ID":"ee971e43-ac12-4c34-892f-a02c57da26e9","Type":"ContainerDied","Data":"72f8b05cdb1377323cec67a55cd592316735636f351258ddc7cc20e305b5d3d8"}
Nov 02 18:38:57.619477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.619330    4956 scope.go:122] "RemoveContainer" containerID="5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9"
Nov 02 18:38:57.630790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.630770    4956 scope.go:122] "RemoveContainer" containerID="b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd"
Nov 02 18:38:57.640179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.640111    4956 scope.go:122] "RemoveContainer" containerID="5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9"
Nov 02 18:38:57.640435 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:38:57.640416    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9\": container with ID starting with 5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9 not found: ID does not exist" containerID="5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9"
Nov 02 18:38:57.640506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.640444    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9"} err="failed to get container status \"5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9\": rpc error: code = NotFound desc = could not find container \"5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9\": container with ID starting with 5519e5ebde3308130e46e40bd2b256fbec1dc14e0d45a08b068879c85eed77d9 not found: ID does not exist"
Nov 02 18:38:57.640506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.640466    4956 scope.go:122] "RemoveContainer" containerID="b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd"
Nov 02 18:38:57.640732 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:38:57.640717    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd\": container with ID starting with b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd not found: ID does not exist" containerID="b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd"
Nov 02 18:38:57.640785 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.640735    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd"} err="failed to get container status \"b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd\": rpc error: code = NotFound desc = could not find container \"b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd\": container with ID starting with b749b8bbd082311046755771f98f7a7edc3997db19f2100dc41b95dd5bc4d1cd not found: ID does not exist"
Nov 02 18:38:57.783229 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.783197    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"]
Nov 02 18:38:57.786755 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.786731    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-608907-openshift-default-6767cf965-rlld4"]
Nov 02 18:38:57.865411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:38:57.865378    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" path="/var/lib/kubelet/pods/ee971e43-ac12-4c34-892f-a02c57da26e9/volumes"
Nov 02 18:38:59.053683 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:38:59.053648    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:39:06.638310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:39:06.638264    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:39:06.639023 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:39:06.638988    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:39:29.053756 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:39:29.053719    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:39:51.290895 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:39:51.290862    4956 scope.go:122] "RemoveContainer" containerID="16226edbaf47551296f71710632882607a25594c0842ee1cd66c557257605305"
Nov 02 18:39:59.053618 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:39:59.053568    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:40:29.054016 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:40:29.053977    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:40:35.900101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:40:35.900063    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:40:35.900787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:40:35.900751    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:40:51.321774 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:40:51.321737    4956 scope.go:122] "RemoveContainer" containerID="fbae76810d89483ac9f8af18f4e090dfec239f53effed634027084d3dafc8979"
Nov 02 18:40:51.331494 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:40:51.331473    4956 scope.go:122] "RemoveContainer" containerID="34bef367bedd2334570abfeff8fabf731af923ea9866213b23f8ac7ebf64f074"
Nov 02 18:40:59.053974 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:40:59.053936    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:41:01.807319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.807274    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"]
Nov 02 18:41:01.808186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808165    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5530c74a-b5e4-46fe-a7cc-0907e34b1afc" containerName="payload-pre-processing"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808186    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5530c74a-b5e4-46fe-a7cc-0907e34b1afc" containerName="payload-pre-processing"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808222    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808230    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808242    4956 cpu_manager.go:414] "containerMap: removing container" podUID="21e55d78-893c-4f4b-af11-41b397777898" containerName="payload-processing"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808249    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e55d78-893c-4f4b-af11-41b397777898" containerName="payload-processing"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808276    4956 cpu_manager.go:414] "containerMap: removing container" podUID="69e6b835-5238-4380-a558-c23cdeb8fbe3" containerName="payload-processing"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808284    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e6b835-5238-4380-a558-c23cdeb8fbe3" containerName="payload-processing"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808295    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy"
Nov 02 18:41:01.808308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808302    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808320    4956 cpu_manager.go:414] "containerMap: removing container" podUID="780d4861-313a-4b69-bf27-ce324a57639a" containerName="maas-api"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808328    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780d4861-313a-4b69-bf27-ce324a57639a" containerName="maas-api"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808486    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5530c74a-b5e4-46fe-a7cc-0907e34b1afc" containerName="payload-pre-processing"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808506    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808518    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e55d78-893c-4f4b-af11-41b397777898" containerName="payload-processing"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808532    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e6b835-5238-4380-a558-c23cdeb8fbe3" containerName="payload-processing"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808540    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee971e43-ac12-4c34-892f-a02c57da26e9" containerName="istio-proxy"
Nov 02 18:41:01.808766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.808548    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="780d4861-313a-4b69-bf27-ce324a57639a" containerName="maas-api"
Nov 02 18:41:01.812148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.812110    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:01.870326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.870294    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"]
Nov 02 18:41:01.914097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.914067    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4fe24d\"/\"kube-root-ca.crt\""
Nov 02 18:41:01.914306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.914108    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4fe24d\"/\"default-dockercfg-4x4jl\""
Nov 02 18:41:01.914306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.914107    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4fe24d\"/\"openshift-service-ca.crt\""
Nov 02 18:41:01.914306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.914113    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4fe24d\"/\"test-model-4fe24d-kserve-self-signed-certs\""
Nov 02 18:41:01.972404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.972368    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-home\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:01.972880 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.972814    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-model-cache\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:01.972880 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.972868    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31062a2-2f20-4e13-9133-9313d8f00f30-tls-certs\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:01.973076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.972913    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj25q\" (UniqueName: \"kubernetes.io/projected/b31062a2-2f20-4e13-9133-9313d8f00f30-kube-api-access-vj25q\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:01.973173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.973079    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-dshm\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:01.973173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:01.973141    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-tmp-dir\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.074713 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.074627    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-home\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.074883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.074719    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-model-cache\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.074883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.074743    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31062a2-2f20-4e13-9133-9313d8f00f30-tls-certs\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.074883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.074767    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj25q\" (UniqueName: \"kubernetes.io/projected/b31062a2-2f20-4e13-9133-9313d8f00f30-kube-api-access-vj25q\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.074883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.074841    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-dshm\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.074883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.074874    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-tmp-dir\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.075236 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.075070    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-home\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.075236 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.075173    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-model-cache\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.075327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.075268    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-tmp-dir\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.077278 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.077252    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-dshm\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.077597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.077581    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31062a2-2f20-4e13-9133-9313d8f00f30-tls-certs\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.133160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.133087    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj25q\" (UniqueName: \"kubernetes.io/projected/b31062a2-2f20-4e13-9133-9313d8f00f30-kube-api-access-vj25q\") pod \"test-model-4fe24d-kserve-5d6855457c-p28tq\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") " pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.424785 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.424704    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:02.802111 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:02.802084    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"]
Nov 02 18:41:02.803020 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:41:02.802993    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb31062a2_2f20_4e13_9133_9313d8f00f30.slice/crio-29500b28c9bb52f561dc144e517c8c8674ff2aa94c7770536b5539b9dc66fd7b WatchSource:0}: Error finding container 29500b28c9bb52f561dc144e517c8c8674ff2aa94c7770536b5539b9dc66fd7b: Status 404 returned error can't find the container with id 29500b28c9bb52f561dc144e517c8c8674ff2aa94c7770536b5539b9dc66fd7b
Nov 02 18:41:03.551781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:03.551741    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq" event={"ID":"b31062a2-2f20-4e13-9133-9313d8f00f30","Type":"ContainerStarted","Data":"7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c"}
Nov 02 18:41:03.552235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:03.551788    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq" event={"ID":"b31062a2-2f20-4e13-9133-9313d8f00f30","Type":"ContainerStarted","Data":"29500b28c9bb52f561dc144e517c8c8674ff2aa94c7770536b5539b9dc66fd7b"}
Nov 02 18:41:03.666857 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:03.666802    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq" podStartSLOduration=2.666786971 podStartE2EDuration="2.666786971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:03.665431449 +0000 UTC m=+4693.922462070" watchObservedRunningTime="2026-11-02 18:41:03.666786971 +0000 UTC m=+4693.923817590"
Nov 02 18:41:04.424982 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:04.424950    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:07.462276 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.462241    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"]
Nov 02 18:41:07.466451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.466433    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.569327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.569297    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"rate-test-model-f7aa2e-kserve-self-signed-certs\""
Nov 02 18:41:07.569536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.569400    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"kube-root-ca.crt\""
Nov 02 18:41:07.569536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.569428    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"default-dockercfg-jzt6h\""
Nov 02 18:41:07.569536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.569470    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"openshift-service-ca.crt\""
Nov 02 18:41:07.572840 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.572811    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"]
Nov 02 18:41:07.643609 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.643573    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-home\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.643790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.643614    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tmp-dir\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.643790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.643742    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-dshm\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.643868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.643804    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfvl\" (UniqueName: \"kubernetes.io/projected/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-kube-api-access-pcfvl\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.643868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.643827    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tls-certs\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.643868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.643854    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-model-cache\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.744929    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-home\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.744971    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tmp-dir\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.745034    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-dshm\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.745111    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfvl\" (UniqueName: \"kubernetes.io/projected/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-kube-api-access-pcfvl\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.745178    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tls-certs\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.745216    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-model-cache\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745482 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.745440    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-home\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745572 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.745544    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tmp-dir\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.745644 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.745618    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-model-cache\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.747766 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.747738    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-dshm\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.747992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.747971    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tls-certs\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:07.801706 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:07.801674    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfvl\" (UniqueName: \"kubernetes.io/projected/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-kube-api-access-pcfvl\") pod \"rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:08.079739 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:08.079651    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:08.454919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:08.454886    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"]
Nov 02 18:41:08.456196 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:41:08.456171    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d96b340_aed6_4f2e_9387_0f8fa9a27f33.slice/crio-58bc3b24ba45dc577974b76ddfb3aefb876d24812b2830edec14cee3ee030254 WatchSource:0}: Error finding container 58bc3b24ba45dc577974b76ddfb3aefb876d24812b2830edec14cee3ee030254: Status 404 returned error can't find the container with id 58bc3b24ba45dc577974b76ddfb3aefb876d24812b2830edec14cee3ee030254
Nov 02 18:41:08.587006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:08.586968    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb" event={"ID":"2d96b340-aed6-4f2e-9387-0f8fa9a27f33","Type":"ContainerStarted","Data":"4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431"}
Nov 02 18:41:08.587445 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:08.587015    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb" event={"ID":"2d96b340-aed6-4f2e-9387-0f8fa9a27f33","Type":"ContainerStarted","Data":"58bc3b24ba45dc577974b76ddfb3aefb876d24812b2830edec14cee3ee030254"}
Nov 02 18:41:08.700935 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:08.700875    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb" podStartSLOduration=1.7008582589999999 podStartE2EDuration="1.700858259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:08.700543247 +0000 UTC m=+4698.957573870" watchObservedRunningTime="2026-11-02 18:41:08.700858259 +0000 UTC m=+4698.957888879"
Nov 02 18:41:09.080579 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:09.080531    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:12.424944 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:12.424906    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:12.438339 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:12.438303    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:12.621747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:12.621719    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:41:18.080756 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:18.080723    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:18.093824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:18.093797    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:18.661329 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:18.661304    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:41:29.054066 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:41:29.054030    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:41:51.361324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:41:51.361289    4956 scope.go:122] "RemoveContainer" containerID="d28bf54faca708ee7745c451683cdc85604a75256f9e39c0d8d7671a49665463"
Nov 02 18:41:59.053285 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:41:59.053247    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:42:01.335606 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.335570    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"]
Nov 02 18:42:01.340328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.340302    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.395501 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.395464    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"]
Nov 02 18:42:01.419199 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.419157    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vp8\" (UniqueName: \"kubernetes.io/projected/1422e817-b4ec-47fd-849b-b368859b9dfa-kube-api-access-l2vp8\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.419412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.419223    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-model-cache\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.419412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.419327    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1422e817-b4ec-47fd-849b-b368859b9dfa-tls-certs\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.419519 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.419470    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-home\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.419569 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.419525    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-dshm\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.419706 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.419685    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-tmp-dir\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.440604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.440565    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"kube-root-ca.crt\""
Nov 02 18:42:01.440787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.440623    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"default-dockercfg-cc5b7\""
Nov 02 18:42:01.440787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.440657    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"rate-test-model-2a1bcc-kserve-self-signed-certs\""
Nov 02 18:42:01.440787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.440667    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2a1bcc\"/\"openshift-service-ca.crt\""
Nov 02 18:42:01.521091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521044    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vp8\" (UniqueName: \"kubernetes.io/projected/1422e817-b4ec-47fd-849b-b368859b9dfa-kube-api-access-l2vp8\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521113    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-model-cache\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521185    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1422e817-b4ec-47fd-849b-b368859b9dfa-tls-certs\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521333 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521297    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-home\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521533 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521356    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-dshm\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521533 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521509    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-tmp-dir\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521644 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521620    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-model-cache\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521898    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-tmp-dir\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.521927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.521914    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-home\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.524493 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.524464    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-dshm\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.524615 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.524598    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1422e817-b4ec-47fd-849b-b368859b9dfa-tls-certs\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.577970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.577941    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vp8\" (UniqueName: \"kubernetes.io/projected/1422e817-b4ec-47fd-849b-b368859b9dfa-kube-api-access-l2vp8\") pod \"rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") " pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:01.654257 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:01.654167    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:02.035991 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:02.035958    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"]
Nov 02 18:42:02.037179 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:42:02.037149    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1422e817_b4ec_47fd_849b_b368859b9dfa.slice/crio-f89ceed804307b1b3bdd677c22afdd7391c60c91b21e0af24c8e69f4ced815fc WatchSource:0}: Error finding container f89ceed804307b1b3bdd677c22afdd7391c60c91b21e0af24c8e69f4ced815fc: Status 404 returned error can't find the container with id f89ceed804307b1b3bdd677c22afdd7391c60c91b21e0af24c8e69f4ced815fc
Nov 02 18:42:02.937778 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:02.937743    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml" event={"ID":"1422e817-b4ec-47fd-849b-b368859b9dfa","Type":"ContainerStarted","Data":"531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b"}
Nov 02 18:42:02.938314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:02.937789    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml" event={"ID":"1422e817-b4ec-47fd-849b-b368859b9dfa","Type":"ContainerStarted","Data":"f89ceed804307b1b3bdd677c22afdd7391c60c91b21e0af24c8e69f4ced815fc"}
Nov 02 18:42:03.054467 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:03.054413    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml" podStartSLOduration=2.054398126 podStartE2EDuration="2.054398126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:42:03.051367666 +0000 UTC m=+4753.308398287" watchObservedRunningTime="2026-11-02 18:42:03.054398126 +0000 UTC m=+4753.311428757"
Nov 02 18:42:03.654399 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:03.654356    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:11.654408 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:11.654374    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:11.667657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:11.667621    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:12.006473 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:12.006445    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:42:29.054512 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:42:29.054476    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:42:50.580696 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:50.580663    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:42:50.588335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:42:50.588303    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:42:59.054222 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:42:59.054185    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:43:12.346733 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.346699    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"]
Nov 02 18:43:12.351013 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.350982    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.433584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.433538    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d564e0b-5e57-4905-8209-548633dd9925-tls-certs\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.433584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.433587    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cts4m\" (UniqueName: \"kubernetes.io/projected/4d564e0b-5e57-4905-8209-548633dd9925-kube-api-access-cts4m\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.433847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.433678    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-model-cache\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.433847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.433728    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-tmp-dir\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.433847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.433796    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-dshm\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.433847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.433848    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-home\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.454141 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.454094    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"openshift-service-ca.crt\""
Nov 02 18:43:12.454141 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.454094    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"kube-root-ca.crt\""
Nov 02 18:43:12.454368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.454111    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"sub-test-model-0ad26a-kserve-self-signed-certs\""
Nov 02 18:43:12.454368 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.454328    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"default-dockercfg-cmd9m\""
Nov 02 18:43:12.457537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.457515    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"]
Nov 02 18:43:12.535153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535107    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d564e0b-5e57-4905-8209-548633dd9925-tls-certs\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535157    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cts4m\" (UniqueName: \"kubernetes.io/projected/4d564e0b-5e57-4905-8209-548633dd9925-kube-api-access-cts4m\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535181    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-model-cache\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535203    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-tmp-dir\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535228    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-dshm\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535437 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535347    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-home\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535692    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-model-cache\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535759 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535727    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-home\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.535801 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.535774    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-tmp-dir\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.537755 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.537726    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-dshm\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.538046 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.538024    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d564e0b-5e57-4905-8209-548633dd9925-tls-certs\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.594831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.594801    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cts4m\" (UniqueName: \"kubernetes.io/projected/4d564e0b-5e57-4905-8209-548633dd9925-kube-api-access-cts4m\") pod \"sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:12.664403 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:12.664301    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:13.040604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:13.040572    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"]
Nov 02 18:43:13.042659 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:43:13.042633    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d564e0b_5e57_4905_8209_548633dd9925.slice/crio-566b1644c1162efd7d4e4f7c1ecb2dc4d154f9cb667c39914d34014822d6160f WatchSource:0}: Error finding container 566b1644c1162efd7d4e4f7c1ecb2dc4d154f9cb667c39914d34014822d6160f: Status 404 returned error can't find the container with id 566b1644c1162efd7d4e4f7c1ecb2dc4d154f9cb667c39914d34014822d6160f
Nov 02 18:43:13.400080 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:13.399980    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz" event={"ID":"4d564e0b-5e57-4905-8209-548633dd9925","Type":"ContainerStarted","Data":"84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f"}
Nov 02 18:43:13.400080 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:13.400023    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz" event={"ID":"4d564e0b-5e57-4905-8209-548633dd9925","Type":"ContainerStarted","Data":"566b1644c1162efd7d4e4f7c1ecb2dc4d154f9cb667c39914d34014822d6160f"}
Nov 02 18:43:13.514613 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:13.514558    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz" podStartSLOduration=1.514543899 podStartE2EDuration="1.514543899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:43:13.511738215 +0000 UTC m=+4823.768768848" watchObservedRunningTime="2026-11-02 18:43:13.514543899 +0000 UTC m=+4823.771574518"
Nov 02 18:43:13.665484 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:13.665401    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:22.664712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:22.664675    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:22.677723 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:22.677694    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:23.478568 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:43:23.478533    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:43:29.055271 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:43:29.055155    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:43:59.053926 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:43:59.053889    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:44:29.053320 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:44:29.053282    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:44:59.053987 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:44:59.053948    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:45:00.195068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.195029    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"]
Nov 02 18:45:00.199448 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.199410    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.204939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.204904    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"]
Nov 02 18:45:00.209218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.209198    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.224023 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.223996    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"]
Nov 02 18:45:00.228328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.228308    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.243910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.243885    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"]
Nov 02 18:45:00.247899 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.247880    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.303800 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.303768    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81069da6-08c5-4bc5-9f40-23e5cf6d329f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.303999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.303883    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97ec46c-c120-48b4-b173-55753b3c8d17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.303999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.303936    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqrt\" (UniqueName: \"kubernetes.io/projected/8315b276-dffd-4c8a-a3cc-1803f9b134e8-kube-api-access-2bqrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.303999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.303968    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2nz\" (UniqueName: \"kubernetes.io/projected/c97ec46c-c120-48b4-b173-55753b3c8d17-kube-api-access-7m2nz\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.304218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.304037    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8315b276-dffd-4c8a-a3cc-1803f9b134e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.304218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.304188    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6qt6\" (UniqueName: \"kubernetes.io/projected/81069da6-08c5-4bc5-9f40-23e5cf6d329f-kube-api-access-g6qt6\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.304334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.304245    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.304334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.304275    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrck\" (UniqueName: \"kubernetes.io/projected/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-kube-api-access-ftrck\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.326936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.326903    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"]
Nov 02 18:45:00.327199 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.327182    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:45:00.372936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.372888    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"]
Nov 02 18:45:00.378217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.378077    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"]
Nov 02 18:45:00.385208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.384319    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"]
Nov 02 18:45:00.405013 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.404978    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.405013 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.405011    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrck\" (UniqueName: \"kubernetes.io/projected/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-kube-api-access-ftrck\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.405296 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.405162    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81069da6-08c5-4bc5-9f40-23e5cf6d329f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.405296 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.405232    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97ec46c-c120-48b4-b173-55753b3c8d17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.405296 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.405275    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqrt\" (UniqueName: \"kubernetes.io/projected/8315b276-dffd-4c8a-a3cc-1803f9b134e8-kube-api-access-2bqrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.405479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.405315    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2nz\" (UniqueName: \"kubernetes.io/projected/c97ec46c-c120-48b4-b173-55753b3c8d17-kube-api-access-7m2nz\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.405479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.405363    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8315b276-dffd-4c8a-a3cc-1803f9b134e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.405479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.405397    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6qt6\" (UniqueName: \"kubernetes.io/projected/81069da6-08c5-4bc5-9f40-23e5cf6d329f-kube-api-access-g6qt6\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.406088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.406062    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.406232 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.406207    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97ec46c-c120-48b4-b173-55753b3c8d17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.406232 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.406215    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8315b276-dffd-4c8a-a3cc-1803f9b134e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.406731 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.406704    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81069da6-08c5-4bc5-9f40-23e5cf6d329f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.461462 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.461433    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2nz\" (UniqueName: \"kubernetes.io/projected/c97ec46c-c120-48b4-b173-55753b3c8d17-kube-api-access-7m2nz\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.461614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.461576    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrck\" (UniqueName: \"kubernetes.io/projected/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-kube-api-access-ftrck\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.461657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.461615    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6qt6\" (UniqueName: \"kubernetes.io/projected/81069da6-08c5-4bc5-9f40-23e5cf6d329f-kube-api-access-g6qt6\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.461657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.461579    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqrt\" (UniqueName: \"kubernetes.io/projected/8315b276-dffd-4c8a-a3cc-1803f9b134e8-kube-api-access-2bqrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.515504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.515473    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:00.526640 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.525651    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:00.544154 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.544110    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:00.561687 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.561653    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:00.910326 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.910293    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"]
Nov 02 18:45:00.910926 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:45:00.910899    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8315b276_dffd_4c8a_a3cc_1803f9b134e8.slice/crio-50bd74340f4fac5951e34ebedcd394dd15f1ac882dc4307b4a8cf84311e24dfd WatchSource:0}: Error finding container 50bd74340f4fac5951e34ebedcd394dd15f1ac882dc4307b4a8cf84311e24dfd: Status 404 returned error can't find the container with id 50bd74340f4fac5951e34ebedcd394dd15f1ac882dc4307b4a8cf84311e24dfd
Nov 02 18:45:00.959343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.959104    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"]
Nov 02 18:45:00.959343 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:45:00.959271    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81069da6_08c5_4bc5_9f40_23e5cf6d329f.slice/crio-ff266f2badf49b46243ed865d116f8828a2b3e2385ccb69f0c7a97fd79bf972d WatchSource:0}: Error finding container ff266f2badf49b46243ed865d116f8828a2b3e2385ccb69f0c7a97fd79bf972d: Status 404 returned error can't find the container with id ff266f2badf49b46243ed865d116f8828a2b3e2385ccb69f0c7a97fd79bf972d
Nov 02 18:45:00.961657 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:45:00.961604    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f9299d_84b1_4ff6_a763_ae7a8e8a655e.slice/crio-020293f37082759bf6b5b9b3d91c1c5c75f97dbb7ba7ae772bbf2e42c5c66c13 WatchSource:0}: Error finding container 020293f37082759bf6b5b9b3d91c1c5c75f97dbb7ba7ae772bbf2e42c5c66c13: Status 404 returned error can't find the container with id 020293f37082759bf6b5b9b3d91c1c5c75f97dbb7ba7ae772bbf2e42c5c66c13
Nov 02 18:45:00.963057 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.963035    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"]
Nov 02 18:45:00.986446 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:00.983100    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"]
Nov 02 18:45:00.991339 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:45:00.991314    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97ec46c_c120_48b4_b173_55753b3c8d17.slice/crio-4d6575c02b47271d0431641c2725a1af3cf5f9f57a7728b57ec18c34b23245e8 WatchSource:0}: Error finding container 4d6575c02b47271d0431641c2725a1af3cf5f9f57a7728b57ec18c34b23245e8: Status 404 returned error can't find the container with id 4d6575c02b47271d0431641c2725a1af3cf5f9f57a7728b57ec18c34b23245e8
Nov 02 18:45:01.098195 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:01.098158    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" event={"ID":"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e","Type":"ContainerStarted","Data":"020293f37082759bf6b5b9b3d91c1c5c75f97dbb7ba7ae772bbf2e42c5c66c13"}
Nov 02 18:45:01.099372 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:01.099342    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" event={"ID":"8315b276-dffd-4c8a-a3cc-1803f9b134e8","Type":"ContainerStarted","Data":"50bd74340f4fac5951e34ebedcd394dd15f1ac882dc4307b4a8cf84311e24dfd"}
Nov 02 18:45:01.100602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:01.100576    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" event={"ID":"c97ec46c-c120-48b4-b173-55753b3c8d17","Type":"ContainerStarted","Data":"4d6575c02b47271d0431641c2725a1af3cf5f9f57a7728b57ec18c34b23245e8"}
Nov 02 18:45:01.101932 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:01.101911    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" event={"ID":"81069da6-08c5-4bc5-9f40-23e5cf6d329f","Type":"ContainerStarted","Data":"ff266f2badf49b46243ed865d116f8828a2b3e2385ccb69f0c7a97fd79bf972d"}
Nov 02 18:45:02.111523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.111482    4956 generic.go:356] "Generic (PLEG): container finished" podID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerID="2ae2f649bf388707a6fb27533f6f75be6f6b6a42c24af0cd943192b8a8affac1" exitCode=60
Nov 02 18:45:02.111948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.111562    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" event={"ID":"81069da6-08c5-4bc5-9f40-23e5cf6d329f","Type":"ContainerDied","Data":"2ae2f649bf388707a6fb27533f6f75be6f6b6a42c24af0cd943192b8a8affac1"}
Nov 02 18:45:02.111948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.111934    4956 scope.go:122] "RemoveContainer" containerID="2ae2f649bf388707a6fb27533f6f75be6f6b6a42c24af0cd943192b8a8affac1"
Nov 02 18:45:02.113868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.113842    4956 generic.go:356] "Generic (PLEG): container finished" podID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerID="bee615a5936696a052a76bce34da1d035fbb4030d23e7e2d68645e727b006653" exitCode=60
Nov 02 18:45:02.113973 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.113926    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" event={"ID":"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e","Type":"ContainerDied","Data":"bee615a5936696a052a76bce34da1d035fbb4030d23e7e2d68645e727b006653"}
Nov 02 18:45:02.114209 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.114192    4956 scope.go:122] "RemoveContainer" containerID="bee615a5936696a052a76bce34da1d035fbb4030d23e7e2d68645e727b006653"
Nov 02 18:45:02.116155 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.115945    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" event={"ID":"8315b276-dffd-4c8a-a3cc-1803f9b134e8","Type":"ContainerStarted","Data":"635b37a6382bcb19c839ae886df326eb605331a728d01d35454682dd8039c069"}
Nov 02 18:45:02.118053 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.118004    4956 generic.go:356] "Generic (PLEG): container finished" podID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerID="bb5a31bd163593784bf34381cadab7a15b4571ec8ecd9359161b919bd0805bab" exitCode=60
Nov 02 18:45:02.118163 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.118053    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" event={"ID":"c97ec46c-c120-48b4-b173-55753b3c8d17","Type":"ContainerDied","Data":"bb5a31bd163593784bf34381cadab7a15b4571ec8ecd9359161b919bd0805bab"}
Nov 02 18:45:02.118364 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.118345    4956 scope.go:122] "RemoveContainer" containerID="bb5a31bd163593784bf34381cadab7a15b4571ec8ecd9359161b919bd0805bab"
Nov 02 18:45:02.343624 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:02.343574    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" podStartSLOduration=2.343560336 podStartE2EDuration="2.343560336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:02.340302924 +0000 UTC m=+4932.597333547" watchObservedRunningTime="2026-11-02 18:45:02.343560336 +0000 UTC m=+4932.600590956"
Nov 02 18:45:03.130443 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.130409    4956 generic.go:356] "Generic (PLEG): container finished" podID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerID="635b37a6382bcb19c839ae886df326eb605331a728d01d35454682dd8039c069" exitCode=60
Nov 02 18:45:03.130894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.130551    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" event={"ID":"8315b276-dffd-4c8a-a3cc-1803f9b134e8","Type":"ContainerDied","Data":"635b37a6382bcb19c839ae886df326eb605331a728d01d35454682dd8039c069"}
Nov 02 18:45:03.130894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.130814    4956 scope.go:122] "RemoveContainer" containerID="635b37a6382bcb19c839ae886df326eb605331a728d01d35454682dd8039c069"
Nov 02 18:45:03.134713 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.134687    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" event={"ID":"c97ec46c-c120-48b4-b173-55753b3c8d17","Type":"ContainerStarted","Data":"ddb01a369e6a5f25e082307fb5c208bb418740612290e8d11a28342ec18a0c03"}
Nov 02 18:45:03.139375 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.137009    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" event={"ID":"81069da6-08c5-4bc5-9f40-23e5cf6d329f","Type":"ContainerStarted","Data":"c98b1fd6a1c3565dacb5ae8d20bfb62d9209848f3f240f1609739d4552fd76e8"}
Nov 02 18:45:03.139375 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.139080    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" event={"ID":"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e","Type":"ContainerStarted","Data":"5640768a9b6a2ec593956f4868d57d7ac1f40a4fa8a521f58903fa871b6c8f98"}
Nov 02 18:45:03.356198 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.356150    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" podStartSLOduration=3.356105555 podStartE2EDuration="3.356105555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:03.353950429 +0000 UTC m=+4933.610981045" watchObservedRunningTime="2026-11-02 18:45:03.356105555 +0000 UTC m=+4933.613136175"
Nov 02 18:45:03.466426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.466375    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" podStartSLOduration=3.466353267 podStartE2EDuration="3.466353267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:03.46527071 +0000 UTC m=+4933.722301330" watchObservedRunningTime="2026-11-02 18:45:03.466353267 +0000 UTC m=+4933.723383888"
Nov 02 18:45:03.581934 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:03.581878    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" podStartSLOduration=3.581863879 podStartE2EDuration="3.581863879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:03.579685244 +0000 UTC m=+4933.836715875" watchObservedRunningTime="2026-11-02 18:45:03.581863879 +0000 UTC m=+4933.838894530"
Nov 02 18:45:04.147783 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.147749    4956 generic.go:356] "Generic (PLEG): container finished" podID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerID="022625b36e32d05d1bd43afbbcbe8d8e73cf64fe9746153808f34ee9736899c0" exitCode=60
Nov 02 18:45:04.148297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.147836    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" event={"ID":"8315b276-dffd-4c8a-a3cc-1803f9b134e8","Type":"ContainerDied","Data":"022625b36e32d05d1bd43afbbcbe8d8e73cf64fe9746153808f34ee9736899c0"}
Nov 02 18:45:04.148297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.147902    4956 scope.go:122] "RemoveContainer" containerID="635b37a6382bcb19c839ae886df326eb605331a728d01d35454682dd8039c069"
Nov 02 18:45:04.148297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.148263    4956 scope.go:122] "RemoveContainer" containerID="022625b36e32d05d1bd43afbbcbe8d8e73cf64fe9746153808f34ee9736899c0"
Nov 02 18:45:04.148630 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:04.148607    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2_odh-ai-gateway-infra(8315b276-dffd-4c8a-a3cc-1803f9b134e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8"
Nov 02 18:45:04.150005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.149985    4956 generic.go:356] "Generic (PLEG): container finished" podID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerID="ddb01a369e6a5f25e082307fb5c208bb418740612290e8d11a28342ec18a0c03" exitCode=60
Nov 02 18:45:04.150163 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.150105    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" event={"ID":"c97ec46c-c120-48b4-b173-55753b3c8d17","Type":"ContainerDied","Data":"ddb01a369e6a5f25e082307fb5c208bb418740612290e8d11a28342ec18a0c03"}
Nov 02 18:45:04.150365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.150348    4956 scope.go:122] "RemoveContainer" containerID="ddb01a369e6a5f25e082307fb5c208bb418740612290e8d11a28342ec18a0c03"
Nov 02 18:45:04.150663 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:04.150642    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj_odh-ai-gateway-infra(c97ec46c-c120-48b4-b173-55753b3c8d17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17"
Nov 02 18:45:04.152035 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.152016    4956 generic.go:356] "Generic (PLEG): container finished" podID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerID="c98b1fd6a1c3565dacb5ae8d20bfb62d9209848f3f240f1609739d4552fd76e8" exitCode=60
Nov 02 18:45:04.152159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.152089    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" event={"ID":"81069da6-08c5-4bc5-9f40-23e5cf6d329f","Type":"ContainerDied","Data":"c98b1fd6a1c3565dacb5ae8d20bfb62d9209848f3f240f1609739d4552fd76e8"}
Nov 02 18:45:04.152362 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.152345    4956 scope.go:122] "RemoveContainer" containerID="c98b1fd6a1c3565dacb5ae8d20bfb62d9209848f3f240f1609739d4552fd76e8"
Nov 02 18:45:04.152627 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:04.152607    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn_odh-ai-gateway-infra(81069da6-08c5-4bc5-9f40-23e5cf6d329f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f"
Nov 02 18:45:04.154229 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.154211    4956 generic.go:356] "Generic (PLEG): container finished" podID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerID="5640768a9b6a2ec593956f4868d57d7ac1f40a4fa8a521f58903fa871b6c8f98" exitCode=60
Nov 02 18:45:04.154307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.154242    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" event={"ID":"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e","Type":"ContainerDied","Data":"5640768a9b6a2ec593956f4868d57d7ac1f40a4fa8a521f58903fa871b6c8f98"}
Nov 02 18:45:04.154492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.154478    4956 scope.go:122] "RemoveContainer" containerID="5640768a9b6a2ec593956f4868d57d7ac1f40a4fa8a521f58903fa871b6c8f98"
Nov 02 18:45:04.154735 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:04.154719    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9_odh-ai-gateway-infra(e3f9299d-84b1-4ff6-a763-ae7a8e8a655e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e"
Nov 02 18:45:04.161725 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.161708    4956 scope.go:122] "RemoveContainer" containerID="bb5a31bd163593784bf34381cadab7a15b4571ec8ecd9359161b919bd0805bab"
Nov 02 18:45:04.175175 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.175152    4956 scope.go:122] "RemoveContainer" containerID="2ae2f649bf388707a6fb27533f6f75be6f6b6a42c24af0cd943192b8a8affac1"
Nov 02 18:45:04.188496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:04.188477    4956 scope.go:122] "RemoveContainer" containerID="bee615a5936696a052a76bce34da1d035fbb4030d23e7e2d68645e727b006653"
Nov 02 18:45:05.162702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:05.162673    4956 scope.go:122] "RemoveContainer" containerID="c98b1fd6a1c3565dacb5ae8d20bfb62d9209848f3f240f1609739d4552fd76e8"
Nov 02 18:45:05.163271 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:05.163004    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn_odh-ai-gateway-infra(81069da6-08c5-4bc5-9f40-23e5cf6d329f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f"
Nov 02 18:45:05.164267 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:05.164244    4956 scope.go:122] "RemoveContainer" containerID="5640768a9b6a2ec593956f4868d57d7ac1f40a4fa8a521f58903fa871b6c8f98"
Nov 02 18:45:05.164533 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:05.164512    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9_odh-ai-gateway-infra(e3f9299d-84b1-4ff6-a763-ae7a8e8a655e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e"
Nov 02 18:45:05.165762 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:05.165742    4956 scope.go:122] "RemoveContainer" containerID="022625b36e32d05d1bd43afbbcbe8d8e73cf64fe9746153808f34ee9736899c0"
Nov 02 18:45:05.166003 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:05.165980    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2_odh-ai-gateway-infra(8315b276-dffd-4c8a-a3cc-1803f9b134e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8"
Nov 02 18:45:05.167312 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:05.167296    4956 scope.go:122] "RemoveContainer" containerID="ddb01a369e6a5f25e082307fb5c208bb418740612290e8d11a28342ec18a0c03"
Nov 02 18:45:05.167494 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:05.167475    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj_odh-ai-gateway-infra(c97ec46c-c120-48b4-b173-55753b3c8d17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17"
Nov 02 18:45:13.858021 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:13.857975    4956 scope.go:122] "RemoveContainer" containerID="5640768a9b6a2ec593956f4868d57d7ac1f40a4fa8a521f58903fa871b6c8f98"
Nov 02 18:45:14.232777 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:14.232738    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" event={"ID":"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e","Type":"ContainerStarted","Data":"566ebc7be1ba7148f12fb235a0bae0ee47b4d0e6c9f52716b11fbb24feeedb3c"}
Nov 02 18:45:15.242249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:15.242215    4956 generic.go:356] "Generic (PLEG): container finished" podID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerID="566ebc7be1ba7148f12fb235a0bae0ee47b4d0e6c9f52716b11fbb24feeedb3c" exitCode=60
Nov 02 18:45:15.242675 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:15.242278    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" event={"ID":"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e","Type":"ContainerDied","Data":"566ebc7be1ba7148f12fb235a0bae0ee47b4d0e6c9f52716b11fbb24feeedb3c"}
Nov 02 18:45:15.242675 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:15.242344    4956 scope.go:122] "RemoveContainer" containerID="5640768a9b6a2ec593956f4868d57d7ac1f40a4fa8a521f58903fa871b6c8f98"
Nov 02 18:45:15.242804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:15.242751    4956 scope.go:122] "RemoveContainer" containerID="566ebc7be1ba7148f12fb235a0bae0ee47b4d0e6c9f52716b11fbb24feeedb3c"
Nov 02 18:45:15.243171 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:15.243152    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9_odh-ai-gateway-infra(e3f9299d-84b1-4ff6-a763-ae7a8e8a655e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e"
Nov 02 18:45:15.355536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:15.355504    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"]
Nov 02 18:45:16.406379 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.406345    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:16.438403 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.438363    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-maas-api-service-ca\") pod \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") "
Nov 02 18:45:16.438568 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.438502    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-kube-api-access-ftrck\" (UniqueName: \"kubernetes.io/projected/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-kube-api-access-ftrck\") pod \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\" (UID: \"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e\") "
Nov 02 18:45:16.438806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.438782    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-maas-api-service-ca" pod "e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" (UID: "e3f9299d-84b1-4ff6-a763-ae7a8e8a655e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:16.439049 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.439034    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:16.440749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.440728    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-kube-api-access-ftrck" pod "e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" (UID: "e3f9299d-84b1-4ff6-a763-ae7a8e8a655e"). InnerVolumeSpecName "kube-api-access-ftrck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:16.541026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.540945    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftrck\" (UniqueName: \"kubernetes.io/projected/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e-kube-api-access-ftrck\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:16.858338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.858243    4956 scope.go:122] "RemoveContainer" containerID="ddb01a369e6a5f25e082307fb5c208bb418740612290e8d11a28342ec18a0c03"
Nov 02 18:45:16.858519 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:16.858425    4956 scope.go:122] "RemoveContainer" containerID="c98b1fd6a1c3565dacb5ae8d20bfb62d9209848f3f240f1609739d4552fd76e8"
Nov 02 18:45:17.260032 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.259995    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" event={"ID":"81069da6-08c5-4bc5-9f40-23e5cf6d329f","Type":"ContainerStarted","Data":"d868b929aa5eba9d0d69b82808412a4a3195d9dfa4a213dc5437cc17c5a4a3b1"}
Nov 02 18:45:17.261355 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.261333    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"
Nov 02 18:45:17.261355 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.261347    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9" event={"ID":"e3f9299d-84b1-4ff6-a763-ae7a8e8a655e","Type":"ContainerDied","Data":"020293f37082759bf6b5b9b3d91c1c5c75f97dbb7ba7ae772bbf2e42c5c66c13"}
Nov 02 18:45:17.261527 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.261404    4956 scope.go:122] "RemoveContainer" containerID="566ebc7be1ba7148f12fb235a0bae0ee47b4d0e6c9f52716b11fbb24feeedb3c"
Nov 02 18:45:17.485242 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.485206    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"]
Nov 02 18:45:17.492746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.492717    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894085-hswr9"]
Nov 02 18:45:17.857813 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.857788    4956 scope.go:122] "RemoveContainer" containerID="022625b36e32d05d1bd43afbbcbe8d8e73cf64fe9746153808f34ee9736899c0"
Nov 02 18:45:17.866283 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:17.866258    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" path="/var/lib/kubelet/pods/e3f9299d-84b1-4ff6-a763-ae7a8e8a655e/volumes"
Nov 02 18:45:18.270030 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.269997    4956 generic.go:356] "Generic (PLEG): container finished" podID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerID="d868b929aa5eba9d0d69b82808412a4a3195d9dfa4a213dc5437cc17c5a4a3b1" exitCode=60
Nov 02 18:45:18.270263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.270077    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" event={"ID":"81069da6-08c5-4bc5-9f40-23e5cf6d329f","Type":"ContainerDied","Data":"d868b929aa5eba9d0d69b82808412a4a3195d9dfa4a213dc5437cc17c5a4a3b1"}
Nov 02 18:45:18.270263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.270169    4956 scope.go:122] "RemoveContainer" containerID="c98b1fd6a1c3565dacb5ae8d20bfb62d9209848f3f240f1609739d4552fd76e8"
Nov 02 18:45:18.270643 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.270624    4956 scope.go:122] "RemoveContainer" containerID="d868b929aa5eba9d0d69b82808412a4a3195d9dfa4a213dc5437cc17c5a4a3b1"
Nov 02 18:45:18.270996 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:18.270967    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn_odh-ai-gateway-infra(81069da6-08c5-4bc5-9f40-23e5cf6d329f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f"
Nov 02 18:45:18.273697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.273674    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" event={"ID":"8315b276-dffd-4c8a-a3cc-1803f9b134e8","Type":"ContainerStarted","Data":"4c32611d4f1fd88abc64b5b5bb383f436f2356f87869556e787dafdcdd4cbbaa"}
Nov 02 18:45:18.275758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.275737    4956 generic.go:356] "Generic (PLEG): container finished" podID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerID="b956795018aaec915be140de4a4ff29d49d466bc127d503d239a1f6b506a43e2" exitCode=60
Nov 02 18:45:18.275883 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.275804    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" event={"ID":"c97ec46c-c120-48b4-b173-55753b3c8d17","Type":"ContainerDied","Data":"b956795018aaec915be140de4a4ff29d49d466bc127d503d239a1f6b506a43e2"}
Nov 02 18:45:18.276174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.276159    4956 scope.go:122] "RemoveContainer" containerID="b956795018aaec915be140de4a4ff29d49d466bc127d503d239a1f6b506a43e2"
Nov 02 18:45:18.276417 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:18.276401    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj_odh-ai-gateway-infra(c97ec46c-c120-48b4-b173-55753b3c8d17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17"
Nov 02 18:45:18.284621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.284606    4956 scope.go:122] "RemoveContainer" containerID="ddb01a369e6a5f25e082307fb5c208bb418740612290e8d11a28342ec18a0c03"
Nov 02 18:45:18.382795 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:18.382753    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"]
Nov 02 18:45:19.286834 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.286801    4956 generic.go:356] "Generic (PLEG): container finished" podID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerID="4c32611d4f1fd88abc64b5b5bb383f436f2356f87869556e787dafdcdd4cbbaa" exitCode=60
Nov 02 18:45:19.287238 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.286827    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" event={"ID":"8315b276-dffd-4c8a-a3cc-1803f9b134e8","Type":"ContainerDied","Data":"4c32611d4f1fd88abc64b5b5bb383f436f2356f87869556e787dafdcdd4cbbaa"}
Nov 02 18:45:19.287238 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.286898    4956 scope.go:122] "RemoveContainer" containerID="022625b36e32d05d1bd43afbbcbe8d8e73cf64fe9746153808f34ee9736899c0"
Nov 02 18:45:19.287356 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.287292    4956 scope.go:122] "RemoveContainer" containerID="4c32611d4f1fd88abc64b5b5bb383f436f2356f87869556e787dafdcdd4cbbaa"
Nov 02 18:45:19.287642 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:19.287615    4956 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2_odh-ai-gateway-infra(8315b276-dffd-4c8a-a3cc-1803f9b134e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8"
Nov 02 18:45:19.450774 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.450746    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:19.479091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.479056    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81069da6-08c5-4bc5-9f40-23e5cf6d329f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81069da6-08c5-4bc5-9f40-23e5cf6d329f-maas-api-service-ca\") pod \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") "
Nov 02 18:45:19.479306 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.479132    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81069da6-08c5-4bc5-9f40-23e5cf6d329f-kube-api-access-g6qt6\" (UniqueName: \"kubernetes.io/projected/81069da6-08c5-4bc5-9f40-23e5cf6d329f-kube-api-access-g6qt6\") pod \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\" (UID: \"81069da6-08c5-4bc5-9f40-23e5cf6d329f\") "
Nov 02 18:45:19.479505 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.479482    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81069da6-08c5-4bc5-9f40-23e5cf6d329f-maas-api-service-ca" pod "81069da6-08c5-4bc5-9f40-23e5cf6d329f" (UID: "81069da6-08c5-4bc5-9f40-23e5cf6d329f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:19.479733 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.479718    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81069da6-08c5-4bc5-9f40-23e5cf6d329f-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:19.481439 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.481413    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81069da6-08c5-4bc5-9f40-23e5cf6d329f-kube-api-access-g6qt6" pod "81069da6-08c5-4bc5-9f40-23e5cf6d329f" (UID: "81069da6-08c5-4bc5-9f40-23e5cf6d329f"). InnerVolumeSpecName "kube-api-access-g6qt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:19.516226 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.516188    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"]
Nov 02 18:45:19.581190 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.581089    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6qt6\" (UniqueName: \"kubernetes.io/projected/81069da6-08c5-4bc5-9f40-23e5cf6d329f-kube-api-access-g6qt6\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:19.625937 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.625905    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"]
Nov 02 18:45:19.805320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.805290    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:19.884819 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.884778    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97ec46c-c120-48b4-b173-55753b3c8d17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97ec46c-c120-48b4-b173-55753b3c8d17-maas-api-service-ca\") pod \"c97ec46c-c120-48b4-b173-55753b3c8d17\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") "
Nov 02 18:45:19.884982 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.884838    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97ec46c-c120-48b4-b173-55753b3c8d17-kube-api-access-7m2nz\" (UniqueName: \"kubernetes.io/projected/c97ec46c-c120-48b4-b173-55753b3c8d17-kube-api-access-7m2nz\") pod \"c97ec46c-c120-48b4-b173-55753b3c8d17\" (UID: \"c97ec46c-c120-48b4-b173-55753b3c8d17\") "
Nov 02 18:45:19.885103 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.885081    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97ec46c-c120-48b4-b173-55753b3c8d17-maas-api-service-ca" pod "c97ec46c-c120-48b4-b173-55753b3c8d17" (UID: "c97ec46c-c120-48b4-b173-55753b3c8d17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:19.885416 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.885398    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97ec46c-c120-48b4-b173-55753b3c8d17-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:19.887076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.887056    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97ec46c-c120-48b4-b173-55753b3c8d17-kube-api-access-7m2nz" pod "c97ec46c-c120-48b4-b173-55753b3c8d17" (UID: "c97ec46c-c120-48b4-b173-55753b3c8d17"). InnerVolumeSpecName "kube-api-access-7m2nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:19.987079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:19.987042    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2nz\" (UniqueName: \"kubernetes.io/projected/c97ec46c-c120-48b4-b173-55753b3c8d17-kube-api-access-7m2nz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.296292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.296263    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"
Nov 02 18:45:20.296773 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.296295    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj" event={"ID":"c97ec46c-c120-48b4-b173-55753b3c8d17","Type":"ContainerDied","Data":"4d6575c02b47271d0431641c2725a1af3cf5f9f57a7728b57ec18c34b23245e8"}
Nov 02 18:45:20.296773 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.296388    4956 scope.go:122] "RemoveContainer" containerID="b956795018aaec915be140de4a4ff29d49d466bc127d503d239a1f6b506a43e2"
Nov 02 18:45:20.298056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.298031    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"
Nov 02 18:45:20.298056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.298038    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn" event={"ID":"81069da6-08c5-4bc5-9f40-23e5cf6d329f","Type":"ContainerDied","Data":"ff266f2badf49b46243ed865d116f8828a2b3e2385ccb69f0c7a97fd79bf972d"}
Nov 02 18:45:20.307199 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.307178    4956 scope.go:122] "RemoveContainer" containerID="d868b929aa5eba9d0d69b82808412a4a3195d9dfa4a213dc5437cc17c5a4a3b1"
Nov 02 18:45:20.463649 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.463622    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:20.493656 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.493626    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8315b276-dffd-4c8a-a3cc-1803f9b134e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8315b276-dffd-4c8a-a3cc-1803f9b134e8-maas-api-service-ca\") pod \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") "
Nov 02 18:45:20.493833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.493819    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8315b276-dffd-4c8a-a3cc-1803f9b134e8-kube-api-access-2bqrt\" (UniqueName: \"kubernetes.io/projected/8315b276-dffd-4c8a-a3cc-1803f9b134e8-kube-api-access-2bqrt\") pod \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\" (UID: \"8315b276-dffd-4c8a-a3cc-1803f9b134e8\") "
Nov 02 18:45:20.493958 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.493937    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8315b276-dffd-4c8a-a3cc-1803f9b134e8-maas-api-service-ca" pod "8315b276-dffd-4c8a-a3cc-1803f9b134e8" (UID: "8315b276-dffd-4c8a-a3cc-1803f9b134e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:20.494366 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.494347    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8315b276-dffd-4c8a-a3cc-1803f9b134e8-maas-api-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.496053 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.496028    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8315b276-dffd-4c8a-a3cc-1803f9b134e8-kube-api-access-2bqrt" pod "8315b276-dffd-4c8a-a3cc-1803f9b134e8" (UID: "8315b276-dffd-4c8a-a3cc-1803f9b134e8"). InnerVolumeSpecName "kube-api-access-2bqrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:20.571994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.571917    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"]
Nov 02 18:45:20.576642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.576617    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894085-pw8mj"]
Nov 02 18:45:20.596387 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.596360    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bqrt\" (UniqueName: \"kubernetes.io/projected/8315b276-dffd-4c8a-a3cc-1803f9b134e8-kube-api-access-2bqrt\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.685568 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.685406    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"]
Nov 02 18:45:20.688558 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:20.688531    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894085-v4gdn"]
Nov 02 18:45:21.307979 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.307944    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"
Nov 02 18:45:21.308578 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.307978    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2" event={"ID":"8315b276-dffd-4c8a-a3cc-1803f9b134e8","Type":"ContainerDied","Data":"50bd74340f4fac5951e34ebedcd394dd15f1ac882dc4307b4a8cf84311e24dfd"}
Nov 02 18:45:21.308578 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.308053    4956 scope.go:122] "RemoveContainer" containerID="4c32611d4f1fd88abc64b5b5bb383f436f2356f87869556e787dafdcdd4cbbaa"
Nov 02 18:45:21.471594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.471558    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"]
Nov 02 18:45:21.474496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.474472    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894085-ppkm2"]
Nov 02 18:45:21.866180 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.866148    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" path="/var/lib/kubelet/pods/81069da6-08c5-4bc5-9f40-23e5cf6d329f/volumes"
Nov 02 18:45:21.866605 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.866591    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" path="/var/lib/kubelet/pods/8315b276-dffd-4c8a-a3cc-1803f9b134e8/volumes"
Nov 02 18:45:21.866983 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:45:21.866970    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" path="/var/lib/kubelet/pods/c97ec46c-c120-48b4-b173-55753b3c8d17/volumes"
Nov 02 18:45:29.053825 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:29.053784    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:45:59.053160 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:45:59.053110    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:46:04.363061 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:04.362973    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"]
Nov 02 18:46:04.363554 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:04.363241    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" containerID="cri-o://dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2" gracePeriod=30
Nov 02 18:46:08.707264 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:08.707223    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:46:08.707749 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:08.707308    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:46:09.633760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.633689    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:46:09.638245 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.638214    4956 generic.go:356] "Generic (PLEG): container finished" podID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerID="dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2" exitCode=0
Nov 02 18:46:09.638405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.638242    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" event={"ID":"f3cd6c76-392c-4f6d-a654-4a86fa888a47","Type":"ContainerDied","Data":"dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2"}
Nov 02 18:46:09.638405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.638298    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs" event={"ID":"f3cd6c76-392c-4f6d-a654-4a86fa888a47","Type":"ContainerDied","Data":"22ae9ea736008b514c5410b2bdef885984039c2e3e32aa9c1c0abd81471a949f"}
Nov 02 18:46:09.638405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.638325    4956 scope.go:122] "RemoveContainer" containerID="dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2"
Nov 02 18:46:09.638405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.638254    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"
Nov 02 18:46:09.648939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.648921    4956 scope.go:122] "RemoveContainer" containerID="dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2"
Nov 02 18:46:09.649271 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:09.649251    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2\": container with ID starting with dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2 not found: ID does not exist" containerID="dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2"
Nov 02 18:46:09.649314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.649281    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2"} err="failed to get container status \"dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2\": rpc error: code = NotFound desc = could not find container \"dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2\": container with ID starting with dae61171c4ca2f33f2e2570c438cc890e2fe241e7b02dfd6468d868f6a2089a2 not found: ID does not exist"
Nov 02 18:46:09.727488 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727454    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-certs\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.727957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727515    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-data\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.727957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727588    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-podinfo\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.727957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727632    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-kube-api-access-r7s5r\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-kube-api-access-r7s5r\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.727957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727666    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istiod-ca-cert\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.727957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727791    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-envoy\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.727957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727773    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-certs" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:09.727957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727828    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-data" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:09.728384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.727973    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-socket\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.728384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728014    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-token\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-token\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.728384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728045    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-credential-socket\") pod \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\" (UID: \"f3cd6c76-392c-4f6d-a654-4a86fa888a47\") "
Nov 02 18:46:09.728384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728164    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istiod-ca-cert" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:09.728384 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728329    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-socket" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:09.728781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728659    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-credential-socket" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:09.728927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728904    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.729029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728960    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.729029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728977    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.729029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.728993    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.729029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.729008    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.730285 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.730250    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-kube-api-access-r7s5r" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "kube-api-access-r7s5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:09.730441 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.730418    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-envoy" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:09.730906 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.730876    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-token" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:09.731014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.730933    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-podinfo" pod "f3cd6c76-392c-4f6d-a654-4a86fa888a47" (UID: "f3cd6c76-392c-4f6d-a654-4a86fa888a47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:46:09.830806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.830768    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.830806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.830798    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7s5r\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-kube-api-access-r7s5r\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.830806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.830808    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:09.831164 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:09.830819    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cd6c76-392c-4f6d-a654-4a86fa888a47-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:10.054066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:10.054027    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"]
Nov 02 18:46:10.057074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:10.057047    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqnqs"]
Nov 02 18:46:11.885978 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:11.885936    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" path="/var/lib/kubelet/pods/f3cd6c76-392c-4f6d-a654-4a86fa888a47/volumes"
Nov 02 18:46:19.484420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.484384    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"]
Nov 02 18:46:19.484990 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.484601    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" containerID="cri-o://d866a2b44f36b5077f54254e0b2a0a54be0a70bc713606acce22fa67550b33ea" gracePeriod=30
Nov 02 18:46:19.535007 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.534975    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"]
Nov 02 18:46:19.535285 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.535238    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" containerID="cri-o://40ce381ee443d9dc815a4cfa6eefbd17b229bc8fdabe6aa0be00d3f01a56b66b" gracePeriod=30
Nov 02 18:46:19.538160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.538134    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"]
Nov 02 18:46:19.538426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.538401    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" containerID="cri-o://e373e7c36aca97ed73c8cbdee2f32c200121d96c6b41a51f06966ad6d1b05c4d" gracePeriod=30
Nov 02 18:46:19.851075 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.850969    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-46pcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:46:19.851075 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:19.851061    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:46:20.586966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:20.586929    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:46:20.587402 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:20.586999    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:46:22.063983 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:22.063944    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"]
Nov 02 18:46:23.666849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:23.666803    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-txlv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:46:23.667420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:23.666890    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:46:24.749637 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.749588    4956 generic.go:356] "Generic (PLEG): container finished" podID="9631198f-23f9-4749-98e2-84573f52b251" containerID="40ce381ee443d9dc815a4cfa6eefbd17b229bc8fdabe6aa0be00d3f01a56b66b" exitCode=0
Nov 02 18:46:24.750151 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.749707    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" event={"ID":"9631198f-23f9-4749-98e2-84573f52b251","Type":"ContainerDied","Data":"40ce381ee443d9dc815a4cfa6eefbd17b229bc8fdabe6aa0be00d3f01a56b66b"}
Nov 02 18:46:24.751596 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.751569    4956 generic.go:356] "Generic (PLEG): container finished" podID="fdb274a7-029a-463d-98be-a5b61361ed63" containerID="d866a2b44f36b5077f54254e0b2a0a54be0a70bc713606acce22fa67550b33ea" exitCode=0
Nov 02 18:46:24.751757 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.751742    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" event={"ID":"fdb274a7-029a-463d-98be-a5b61361ed63","Type":"ContainerDied","Data":"d866a2b44f36b5077f54254e0b2a0a54be0a70bc713606acce22fa67550b33ea"}
Nov 02 18:46:24.753535 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.753486    4956 generic.go:356] "Generic (PLEG): container finished" podID="aedc2ef6-768a-4464-8292-4363506a59f2" containerID="e373e7c36aca97ed73c8cbdee2f32c200121d96c6b41a51f06966ad6d1b05c4d" exitCode=0
Nov 02 18:46:24.753620 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.753605    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" event={"ID":"aedc2ef6-768a-4464-8292-4363506a59f2","Type":"ContainerDied","Data":"e373e7c36aca97ed73c8cbdee2f32c200121d96c6b41a51f06966ad6d1b05c4d"}
Nov 02 18:46:24.832993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.832959    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:46:24.865576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.865544    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:46:24.876872 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.876844    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:46:24.928317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928282    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb274a7-029a-463d-98be-a5b61361ed63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb274a7-029a-463d-98be-a5b61361ed63-istiod-ca-cert\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.928317 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928321    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-istio-token\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-istio-token\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.928570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928367    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-socket\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.928570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928390    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-kube-api-access-95q7q\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-kube-api-access-95q7q\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.928570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928417    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-certs\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.928570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928434    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-data\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.928570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928510    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-kube-api-access-gw259\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-kube-api-access-gw259\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.928570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928550    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-envoy\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928581    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-credential-socket\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928632    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-kube-api-access-dg5nd\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-kube-api-access-dg5nd\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928669    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdb274a7-029a-463d-98be-a5b61361ed63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb274a7-029a-463d-98be-a5b61361ed63-istio-podinfo\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928696    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-socket\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928724    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-envoy\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928727    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-socket" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928745    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-certs" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928756    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedc2ef6-768a-4464-8292-4363506a59f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedc2ef6-768a-4464-8292-4363506a59f2-istiod-ca-cert\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928789    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb274a7-029a-463d-98be-a5b61361ed63-istiod-ca-cert" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:24.928903 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928811    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-istio-token\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-istio-token\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928909    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-istio-token\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-istio-token\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928928    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-credential-socket" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.928982    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9631198f-23f9-4749-98e2-84573f52b251-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9631198f-23f9-4749-98e2-84573f52b251-istio-podinfo\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929005    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-data" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929024    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-credential-socket\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929066    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-data\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929093    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-certs\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929110    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedc2ef6-768a-4464-8292-4363506a59f2-istiod-ca-cert" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929220    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-data\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929380    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-certs\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929422    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-socket\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929464    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-credential-socket\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929497    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aedc2ef6-768a-4464-8292-4363506a59f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedc2ef6-768a-4464-8292-4363506a59f2-istio-podinfo\") pod \"aedc2ef6-768a-4464-8292-4363506a59f2\" (UID: \"aedc2ef6-768a-4464-8292-4363506a59f2\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929543    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9631198f-23f9-4749-98e2-84573f52b251-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9631198f-23f9-4749-98e2-84573f52b251-istiod-ca-cert\") pod \"9631198f-23f9-4749-98e2-84573f52b251\" (UID: \"9631198f-23f9-4749-98e2-84573f52b251\") "
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929605    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-socket" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.929712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.929619    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-envoy\") pod \"fdb274a7-029a-463d-98be-a5b61361ed63\" (UID: \"fdb274a7-029a-463d-98be-a5b61361ed63\") "
Nov 02 18:46:24.930669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.930431    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb274a7-029a-463d-98be-a5b61361ed63-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:24.930669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.930455    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:24.930669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.930471    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:24.930669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.930485    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:24.930669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.930501    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:24.930669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.930515    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:24.930669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.930531    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedc2ef6-768a-4464-8292-4363506a59f2-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:24.931337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.931307    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-istio-token" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:24.931451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.931400    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdb274a7-029a-463d-98be-a5b61361ed63-istio-podinfo" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:46:24.932471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932062    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-envoy" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.932471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932059    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-envoy" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.932471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932080    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-kube-api-access-gw259" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "kube-api-access-gw259". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:24.932471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932180    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-kube-api-access-95q7q" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "kube-api-access-95q7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:24.932471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932351    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-data" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.932910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932556    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-credential-socket" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.932910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932720    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-data" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.932910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.932897    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-certs" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.933097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.933086    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-socket" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.933190 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.933106    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9631198f-23f9-4749-98e2-84573f52b251-istiod-ca-cert" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:24.933319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.933298    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-certs" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.933510 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.933489    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-credential-socket" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:24.933951 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.933904    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9631198f-23f9-4749-98e2-84573f52b251-istio-podinfo" pod "9631198f-23f9-4749-98e2-84573f52b251" (UID: "9631198f-23f9-4749-98e2-84573f52b251"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:46:24.934255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.934211    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-kube-api-access-dg5nd" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "kube-api-access-dg5nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:24.934359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.934329    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-istio-token" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:24.934359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.934345    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-istio-token" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:24.934504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.934488    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aedc2ef6-768a-4464-8292-4363506a59f2-istio-podinfo" pod "aedc2ef6-768a-4464-8292-4363506a59f2" (UID: "aedc2ef6-768a-4464-8292-4363506a59f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:46:24.935279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:24.935251    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-envoy" pod "fdb274a7-029a-463d-98be-a5b61361ed63" (UID: "fdb274a7-029a-463d-98be-a5b61361ed63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032165    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032197    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9631198f-23f9-4749-98e2-84573f52b251-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032208    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032218    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032227    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032236    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032244    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032252    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032260    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032268    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedc2ef6-768a-4464-8292-4363506a59f2-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.032265 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032276    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9631198f-23f9-4749-98e2-84573f52b251-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032285    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb274a7-029a-463d-98be-a5b61361ed63-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032293    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032301    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95q7q\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-kube-api-access-95q7q\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032312    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw259\" (UniqueName: \"kubernetes.io/projected/9631198f-23f9-4749-98e2-84573f52b251-kube-api-access-gw259\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032323    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedc2ef6-768a-4464-8292-4363506a59f2-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032331    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5nd\" (UniqueName: \"kubernetes.io/projected/fdb274a7-029a-463d-98be-a5b61361ed63-kube-api-access-dg5nd\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032339    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb274a7-029a-463d-98be-a5b61361ed63-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032349    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9631198f-23f9-4749-98e2-84573f52b251-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.033003 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.032358    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedc2ef6-768a-4464-8292-4363506a59f2-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:25.761364 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.761329    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"
Nov 02 18:46:25.761364 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.761358    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb" event={"ID":"aedc2ef6-768a-4464-8292-4363506a59f2","Type":"ContainerDied","Data":"46bba3aea2854f19eb2dcc8c4878d3ea15f5649096d3a31dcf3a98ab234c500a"}
Nov 02 18:46:25.761896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.761411    4956 scope.go:122] "RemoveContainer" containerID="e373e7c36aca97ed73c8cbdee2f32c200121d96c6b41a51f06966ad6d1b05c4d"
Nov 02 18:46:25.763014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.762992    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"
Nov 02 18:46:25.763173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.763015    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5" event={"ID":"9631198f-23f9-4749-98e2-84573f52b251","Type":"ContainerDied","Data":"0bef2c7befa20b05a267f0ee6b0e233799627fb73416fbf3edb02755183386e8"}
Nov 02 18:46:25.765016 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.764998    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"
Nov 02 18:46:25.765159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.765038    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4" event={"ID":"fdb274a7-029a-463d-98be-a5b61361ed63","Type":"ContainerDied","Data":"39bd015949359ec2c178181c00458097c95f114f17717bab047392a966d1f34a"}
Nov 02 18:46:25.774211 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.774180    4956 scope.go:122] "RemoveContainer" containerID="40ce381ee443d9dc815a4cfa6eefbd17b229bc8fdabe6aa0be00d3f01a56b66b"
Nov 02 18:46:25.784393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.784368    4956 scope.go:122] "RemoveContainer" containerID="d866a2b44f36b5077f54254e0b2a0a54be0a70bc713606acce22fa67550b33ea"
Nov 02 18:46:25.930129 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.930085    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"]
Nov 02 18:46:25.933275 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:25.933255    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lkfz5"]
Nov 02 18:46:26.043412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:26.043330    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"]
Nov 02 18:46:26.046860 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:26.046835    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-46pcb"]
Nov 02 18:46:26.155093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:26.155064    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"]
Nov 02 18:46:26.158495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:26.158471    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-txlv4"]
Nov 02 18:46:27.064537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:27.064491    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" podUID="5945f88d-c0bb-46b3-a915-3f0f17b84217" containerName="payload-pre-processing" containerID="cri-o://4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449" gracePeriod=25
Nov 02 18:46:27.865854 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:27.865819    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9631198f-23f9-4749-98e2-84573f52b251" path="/var/lib/kubelet/pods/9631198f-23f9-4749-98e2-84573f52b251/volumes"
Nov 02 18:46:27.866329 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:27.866315    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" path="/var/lib/kubelet/pods/aedc2ef6-768a-4464-8292-4363506a59f2/volumes"
Nov 02 18:46:27.866740 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:27.866726    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" path="/var/lib/kubelet/pods/fdb274a7-029a-463d-98be-a5b61361ed63/volumes"
Nov 02 18:46:29.052943 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:29.052899    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:46:31.157852 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.157817    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7517-bba12e"]
Nov 02 18:46:31.158452 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158436    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158453    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158471    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158476    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158485    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158490    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158498    4956 cpu_manager.go:414] "containerMap: removing container" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158502    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158509    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:31.158516 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158514    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158528    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158532    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158543    4956 cpu_manager.go:414] "containerMap: removing container" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158547    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158557    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158561    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158571    4956 cpu_manager.go:414] "containerMap: removing container" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158575    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158585    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158589    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158595    4956 cpu_manager.go:414] "containerMap: removing container" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158601    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158607    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158611    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158620    4956 cpu_manager.go:414] "containerMap: removing container" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158624    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158634    4956 cpu_manager.go:414] "containerMap: removing container" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158638    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158720    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158728    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158737    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb274a7-029a-463d-98be-a5b61361ed63" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158743    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158748    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158757    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="9631198f-23f9-4749-98e2-84573f52b251" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158762    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158771    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cd6c76-392c-4f6d-a654-4a86fa888a47" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158778    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158783    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158789    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedc2ef6-768a-4464-8292-4363506a59f2" containerName="istio-proxy"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158795    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.158790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158801    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f9299d-84b1-4ff6-a763-ae7a8e8a655e" containerName="cleanup"
Nov 02 18:46:31.159743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.158806    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:31.163987 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.163967    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:31.214425 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.214385    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjtc\" (UniqueName: \"kubernetes.io/projected/8fb81a4f-ad97-4069-870d-d34e13c1f0bb-kube-api-access-gqjtc\") pod \"test-curl-7517-bba12e\" (UID: \"8fb81a4f-ad97-4069-870d-d34e13c1f0bb\") " pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:31.216332 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.216303    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-bba12e"]
Nov 02 18:46:31.262077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.262047    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4gwc\""
Nov 02 18:46:31.315682 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.315646    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjtc\" (UniqueName: \"kubernetes.io/projected/8fb81a4f-ad97-4069-870d-d34e13c1f0bb-kube-api-access-gqjtc\") pod \"test-curl-7517-bba12e\" (UID: \"8fb81a4f-ad97-4069-870d-d34e13c1f0bb\") " pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:31.372604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.372566    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjtc\" (UniqueName: \"kubernetes.io/projected/8fb81a4f-ad97-4069-870d-d34e13c1f0bb-kube-api-access-gqjtc\") pod \"test-curl-7517-bba12e\" (UID: \"8fb81a4f-ad97-4069-870d-d34e13c1f0bb\") " pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:31.476959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.476921    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:31.868959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:31.868934    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-bba12e"]
Nov 02 18:46:31.870452 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:46:31.870423    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fb81a4f_ad97_4069_870d_d34e13c1f0bb.slice/crio-3b4d679394638ce7b95333a2bf39c6c1ddf5fe8764753bc9ff74e65dfe385e04 WatchSource:0}: Error finding container 3b4d679394638ce7b95333a2bf39c6c1ddf5fe8764753bc9ff74e65dfe385e04: Status 404 returned error can't find the container with id 3b4d679394638ce7b95333a2bf39c6c1ddf5fe8764753bc9ff74e65dfe385e04
Nov 02 18:46:32.423879 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:32.423843    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"]
Nov 02 18:46:32.818238 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:32.818198    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-bba12e" event={"ID":"8fb81a4f-ad97-4069-870d-d34e13c1f0bb","Type":"ContainerStarted","Data":"93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d"}
Nov 02 18:46:32.818416 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:32.818250    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-bba12e" event={"ID":"8fb81a4f-ad97-4069-870d-d34e13c1f0bb","Type":"ContainerStarted","Data":"3b4d679394638ce7b95333a2bf39c6c1ddf5fe8764753bc9ff74e65dfe385e04"}
Nov 02 18:46:32.929296 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:32.929243    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7517-bba12e" podStartSLOduration=1.9292274950000001 podStartE2EDuration="1.929227495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:32.928234095 +0000 UTC m=+5023.185264715" watchObservedRunningTime="2026-11-02 18:46:32.929227495 +0000 UTC m=+5023.186258114"
Nov 02 18:46:34.353601 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.353567    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7517-bba12e"]
Nov 02 18:46:34.356698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.356669    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7517-bba12e"]
Nov 02 18:46:34.497758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.497724    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7517-1800eb"]
Nov 02 18:46:34.498350 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498336    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" containerName="test-curl-7517-bba12e"
Nov 02 18:46:34.498420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498351    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" containerName="test-curl-7517-bba12e"
Nov 02 18:46:34.498420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498377    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:34.498420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498386    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8315b276-dffd-4c8a-a3cc-1803f9b134e8" containerName="cleanup"
Nov 02 18:46:34.498420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498402    4956 cpu_manager.go:414] "containerMap: removing container" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:34.498420 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498406    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:34.498590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498486    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="81069da6-08c5-4bc5-9f40-23e5cf6d329f" containerName="cleanup"
Nov 02 18:46:34.498590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498495    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" containerName="test-curl-7517-bba12e"
Nov 02 18:46:34.498590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.498508    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97ec46c-c120-48b4-b173-55753b3c8d17" containerName="cleanup"
Nov 02 18:46:34.501920 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.501904    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:34.554307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.554277    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-1800eb"]
Nov 02 18:46:34.658641 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.658541    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576p6\" (UniqueName: \"kubernetes.io/projected/072b9033-e88a-4b53-a72a-fbbd1a30d782-kube-api-access-576p6\") pod \"test-curl-7517-1800eb\" (UID: \"072b9033-e88a-4b53-a72a-fbbd1a30d782\") " pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:34.760379 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.760337    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576p6\" (UniqueName: \"kubernetes.io/projected/072b9033-e88a-4b53-a72a-fbbd1a30d782-kube-api-access-576p6\") pod \"test-curl-7517-1800eb\" (UID: \"072b9033-e88a-4b53-a72a-fbbd1a30d782\") " pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:34.816607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.816566    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576p6\" (UniqueName: \"kubernetes.io/projected/072b9033-e88a-4b53-a72a-fbbd1a30d782-kube-api-access-576p6\") pod \"test-curl-7517-1800eb\" (UID: \"072b9033-e88a-4b53-a72a-fbbd1a30d782\") " pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:34.832419 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:34.832379    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7517-bba12e" podUID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" containerName="test-curl-7517-bba12e" containerID="cri-o://93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d" gracePeriod=2
Nov 02 18:46:34.883381 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:34.883339    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7517-bba12e\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:35.114493 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:35.114454    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:35.485144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:35.485097    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" podUID="5945f88d-c0bb-46b3-a915-3f0f17b84217" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:46:35.486850 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:35.486830    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-1800eb"]
Nov 02 18:46:35.488722 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:46:35.488696    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod072b9033_e88a_4b53_a72a_fbbd1a30d782.slice/crio-5fcf399e3f9e73c008d74372d5365faaa54803f6989b258d90c7898d55f4ab0a WatchSource:0}: Error finding container 5fcf399e3f9e73c008d74372d5365faaa54803f6989b258d90c7898d55f4ab0a: Status 404 returned error can't find the container with id 5fcf399e3f9e73c008d74372d5365faaa54803f6989b258d90c7898d55f4ab0a
Nov 02 18:46:35.840949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:35.840851    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-1800eb" event={"ID":"072b9033-e88a-4b53-a72a-fbbd1a30d782","Type":"ContainerStarted","Data":"74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f"}
Nov 02 18:46:35.840949 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:35.840891    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-1800eb" event={"ID":"072b9033-e88a-4b53-a72a-fbbd1a30d782","Type":"ContainerStarted","Data":"5fcf399e3f9e73c008d74372d5365faaa54803f6989b258d90c7898d55f4ab0a"}
Nov 02 18:46:35.891794 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:35.891752    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7517-bba12e\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:36.004534 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:36.004477    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7517-1800eb" podStartSLOduration=2.004461526 podStartE2EDuration="2.004461526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:36.001860054 +0000 UTC m=+5026.258890676" watchObservedRunningTime="2026-11-02 18:46:36.004461526 +0000 UTC m=+5026.261492146"
Nov 02 18:46:37.094550 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.094484    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:37.192472 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.192442    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb81a4f-ad97-4069-870d-d34e13c1f0bb-kube-api-access-gqjtc\" (UniqueName: \"kubernetes.io/projected/8fb81a4f-ad97-4069-870d-d34e13c1f0bb-kube-api-access-gqjtc\") pod \"8fb81a4f-ad97-4069-870d-d34e13c1f0bb\" (UID: \"8fb81a4f-ad97-4069-870d-d34e13c1f0bb\") "
Nov 02 18:46:37.194984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.194956    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb81a4f-ad97-4069-870d-d34e13c1f0bb-kube-api-access-gqjtc" pod "8fb81a4f-ad97-4069-870d-d34e13c1f0bb" (UID: "8fb81a4f-ad97-4069-870d-d34e13c1f0bb"). InnerVolumeSpecName "kube-api-access-gqjtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:37.294249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.294208    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjtc\" (UniqueName: \"kubernetes.io/projected/8fb81a4f-ad97-4069-870d-d34e13c1f0bb-kube-api-access-gqjtc\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:37.386782 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.386683    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7517-1800eb"]
Nov 02 18:46:37.390347 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.390299    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7517-1800eb"]
Nov 02 18:46:37.855929 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.855891    4956 generic.go:356] "Generic (PLEG): container finished" podID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" containerID="93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d" exitCode=137
Nov 02 18:46:37.856160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.855939    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-bba12e"
Nov 02 18:46:37.856160 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.855950    4956 scope.go:122] "RemoveContainer" containerID="93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d"
Nov 02 18:46:37.856441 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.856412    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7517-1800eb" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" containerName="test-curl-7517-1800eb" containerID="cri-o://74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f" gracePeriod=2
Nov 02 18:46:37.870465 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.870299    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb81a4f-ad97-4069-870d-d34e13c1f0bb" path="/var/lib/kubelet/pods/8fb81a4f-ad97-4069-870d-d34e13c1f0bb/volumes"
Nov 02 18:46:37.870537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.870495    4956 scope.go:122] "RemoveContainer" containerID="93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d"
Nov 02 18:46:37.870820 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:37.870794    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d\": container with ID starting with 93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d not found: ID does not exist" containerID="93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d"
Nov 02 18:46:37.870936 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:37.870828    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d"} err="failed to get container status \"93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d\": rpc error: code = NotFound desc = could not find container \"93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d\": container with ID starting with 93f5c1822efba307059a977918c9926a593b804006ecb2497e2ba782e2aa985d not found: ID does not exist"
Nov 02 18:46:37.907186 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:37.907145    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7517-1800eb\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:40.105457 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.105431    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:40.156103 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:40.156006    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7517-1800eb\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:40.229860 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.229823    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072b9033-e88a-4b53-a72a-fbbd1a30d782-kube-api-access-576p6\" (UniqueName: \"kubernetes.io/projected/072b9033-e88a-4b53-a72a-fbbd1a30d782-kube-api-access-576p6\") pod \"072b9033-e88a-4b53-a72a-fbbd1a30d782\" (UID: \"072b9033-e88a-4b53-a72a-fbbd1a30d782\") "
Nov 02 18:46:40.232575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.232536    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072b9033-e88a-4b53-a72a-fbbd1a30d782-kube-api-access-576p6" pod "072b9033-e88a-4b53-a72a-fbbd1a30d782" (UID: "072b9033-e88a-4b53-a72a-fbbd1a30d782"). InnerVolumeSpecName "kube-api-access-576p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:40.331687 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.331646    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576p6\" (UniqueName: \"kubernetes.io/projected/072b9033-e88a-4b53-a72a-fbbd1a30d782-kube-api-access-576p6\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:40.880635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.880600    4956 generic.go:356] "Generic (PLEG): container finished" podID="072b9033-e88a-4b53-a72a-fbbd1a30d782" containerID="74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f" exitCode=137
Nov 02 18:46:40.880842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.880643    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:40.880842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.880707    4956 scope.go:122] "RemoveContainer" containerID="74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f"
Nov 02 18:46:40.890896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.890876    4956 scope.go:122] "RemoveContainer" containerID="74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f"
Nov 02 18:46:40.891200 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:40.891178    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f\": container with ID starting with 74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f not found: ID does not exist" containerID="74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f"
Nov 02 18:46:40.891309 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:40.891205    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f"} err="failed to get container status \"74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f\": rpc error: code = NotFound desc = could not find container \"74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f\": container with ID starting with 74758536249b859865f0f844087582e7458286c191ce3e736fd95f309f3cdd5f not found: ID does not exist"
Nov 02 18:46:40.931382 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:40.931339    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7517-1800eb\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:40.981963 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:40.981914    4956 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7517-1800eb\" is forbidden: User \"system:node:ip-10-0-129-72.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-72.us-west-1.compute.internal' and this object" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" pod="opendatahub/test-curl-7517-1800eb"
Nov 02 18:46:41.865912 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:41.865874    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" path="/var/lib/kubelet/pods/072b9033-e88a-4b53-a72a-fbbd1a30d782/volumes"
Nov 02 18:46:45.485774 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:45.485740    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" podUID="5945f88d-c0bb-46b3-a915-3f0f17b84217" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:46:47.348955 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.348921    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:46:47.423821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.423786    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5945f88d-c0bb-46b3-a915-3f0f17b84217-kube-api-access-42gjg\" (UniqueName: \"kubernetes.io/projected/5945f88d-c0bb-46b3-a915-3f0f17b84217-kube-api-access-42gjg\") pod \"5945f88d-c0bb-46b3-a915-3f0f17b84217\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") "
Nov 02 18:46:47.424093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.423915    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5945f88d-c0bb-46b3-a915-3f0f17b84217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5945f88d-c0bb-46b3-a915-3f0f17b84217-plugins-config-volume\") pod \"5945f88d-c0bb-46b3-a915-3f0f17b84217\" (UID: \"5945f88d-c0bb-46b3-a915-3f0f17b84217\") "
Nov 02 18:46:47.424367 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.424340    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5945f88d-c0bb-46b3-a915-3f0f17b84217-plugins-config-volume" pod "5945f88d-c0bb-46b3-a915-3f0f17b84217" (UID: "5945f88d-c0bb-46b3-a915-3f0f17b84217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:47.426163 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.426140    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5945f88d-c0bb-46b3-a915-3f0f17b84217-kube-api-access-42gjg" pod "5945f88d-c0bb-46b3-a915-3f0f17b84217" (UID: "5945f88d-c0bb-46b3-a915-3f0f17b84217"). InnerVolumeSpecName "kube-api-access-42gjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:47.447097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.447053    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq" podUID="b31062a2-2f20-4e13-9133-9313d8f00f30" containerName="main" containerID="cri-o://7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c" gracePeriod=45
Nov 02 18:46:47.525481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.525442    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5945f88d-c0bb-46b3-a915-3f0f17b84217-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.525481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.525476    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gjg\" (UniqueName: \"kubernetes.io/projected/5945f88d-c0bb-46b3-a915-3f0f17b84217-kube-api-access-42gjg\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.611469 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.611373    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq" podUID="b31062a2-2f20-4e13-9133-9313d8f00f30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": dial tcp 10.132.0.191:8000: connect: connection refused"
Nov 02 18:46:47.704824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.704794    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:46:47.728041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728009    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31062a2-2f20-4e13-9133-9313d8f00f30-tls-certs\" (UniqueName: \"kubernetes.io/secret/b31062a2-2f20-4e13-9133-9313d8f00f30-tls-certs\") pod \"b31062a2-2f20-4e13-9133-9313d8f00f30\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") "
Nov 02 18:46:47.728261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728135    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-dshm\") pod \"b31062a2-2f20-4e13-9133-9313d8f00f30\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") "
Nov 02 18:46:47.728261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728207    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31062a2-2f20-4e13-9133-9313d8f00f30-kube-api-access-vj25q\" (UniqueName: \"kubernetes.io/projected/b31062a2-2f20-4e13-9133-9313d8f00f30-kube-api-access-vj25q\") pod \"b31062a2-2f20-4e13-9133-9313d8f00f30\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") "
Nov 02 18:46:47.728381 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728319    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-model-cache\") pod \"b31062a2-2f20-4e13-9133-9313d8f00f30\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") "
Nov 02 18:46:47.728381 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728351    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-tmp-dir\") pod \"b31062a2-2f20-4e13-9133-9313d8f00f30\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") "
Nov 02 18:46:47.728487 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728383    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-home\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-home\") pod \"b31062a2-2f20-4e13-9133-9313d8f00f30\" (UID: \"b31062a2-2f20-4e13-9133-9313d8f00f30\") "
Nov 02 18:46:47.728625 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728597    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-tmp-dir" pod "b31062a2-2f20-4e13-9133-9313d8f00f30" (UID: "b31062a2-2f20-4e13-9133-9313d8f00f30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:47.728748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728659    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-model-cache" pod "b31062a2-2f20-4e13-9133-9313d8f00f30" (UID: "b31062a2-2f20-4e13-9133-9313d8f00f30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:47.728889 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728804    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-home" pod "b31062a2-2f20-4e13-9133-9313d8f00f30" (UID: "b31062a2-2f20-4e13-9133-9313d8f00f30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:47.728984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728965    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.729051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.728989    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.729051 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.729003    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.730822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.730798    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31062a2-2f20-4e13-9133-9313d8f00f30-tls-certs" pod "b31062a2-2f20-4e13-9133-9313d8f00f30" (UID: "b31062a2-2f20-4e13-9133-9313d8f00f30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:46:47.730924 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.730894    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-dshm" pod "b31062a2-2f20-4e13-9133-9313d8f00f30" (UID: "b31062a2-2f20-4e13-9133-9313d8f00f30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:46:47.731045 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.731021    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31062a2-2f20-4e13-9133-9313d8f00f30-kube-api-access-vj25q" pod "b31062a2-2f20-4e13-9133-9313d8f00f30" (UID: "b31062a2-2f20-4e13-9133-9313d8f00f30"). InnerVolumeSpecName "kube-api-access-vj25q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:47.830319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.830282    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31062a2-2f20-4e13-9133-9313d8f00f30-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.830319 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.830321    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31062a2-2f20-4e13-9133-9313d8f00f30-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.830593 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.830336    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj25q\" (UniqueName: \"kubernetes.io/projected/b31062a2-2f20-4e13-9133-9313d8f00f30-kube-api-access-vj25q\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:47.929930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.929889    4956 generic.go:356] "Generic (PLEG): container finished" podID="5945f88d-c0bb-46b3-a915-3f0f17b84217" containerID="4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449" exitCode=0
Nov 02 18:46:47.930143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.929938    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"
Nov 02 18:46:47.930143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.929966    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" event={"ID":"5945f88d-c0bb-46b3-a915-3f0f17b84217","Type":"ContainerDied","Data":"4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449"}
Nov 02 18:46:47.930143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.930027    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6" event={"ID":"5945f88d-c0bb-46b3-a915-3f0f17b84217","Type":"ContainerDied","Data":"b29ca0e09170772d7ae6bb33b1521854062a70c1dcad6066029d1dd85c98ee13"}
Nov 02 18:46:47.930143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.930054    4956 scope.go:122] "RemoveContainer" containerID="4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449"
Nov 02 18:46:47.931599 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.931578    4956 generic.go:356] "Generic (PLEG): container finished" podID="b31062a2-2f20-4e13-9133-9313d8f00f30" containerID="7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c" exitCode=0
Nov 02 18:46:47.931811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.931790    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq" event={"ID":"b31062a2-2f20-4e13-9133-9313d8f00f30","Type":"ContainerDied","Data":"7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c"}
Nov 02 18:46:47.931879 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.931834    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq" event={"ID":"b31062a2-2f20-4e13-9133-9313d8f00f30","Type":"ContainerDied","Data":"29500b28c9bb52f561dc144e517c8c8674ff2aa94c7770536b5539b9dc66fd7b"}
Nov 02 18:46:47.931922 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.931899    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"
Nov 02 18:46:47.940863 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.940834    4956 scope.go:122] "RemoveContainer" containerID="4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449"
Nov 02 18:46:47.941241 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:47.941221    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449\": container with ID starting with 4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449 not found: ID does not exist" containerID="4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449"
Nov 02 18:46:47.941323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.941252    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449"} err="failed to get container status \"4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449\": rpc error: code = NotFound desc = could not find container \"4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449\": container with ID starting with 4311db1936ace637b2893cf2a120e750c7d2c50de169f2fa3adaa8250db8f449 not found: ID does not exist"
Nov 02 18:46:47.941323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.941274    4956 scope.go:122] "RemoveContainer" containerID="7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c"
Nov 02 18:46:47.951059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.951037    4956 scope.go:122] "RemoveContainer" containerID="7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c"
Nov 02 18:46:47.951407 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:47.951386    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c\": container with ID starting with 7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c not found: ID does not exist" containerID="7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c"
Nov 02 18:46:47.951497 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:47.951420    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c"} err="failed to get container status \"7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c\": rpc error: code = NotFound desc = could not find container \"7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c\": container with ID starting with 7b1353ba5eee41db964ce89f65c95c4503d88eb46c314bff91655c666564040c not found: ID does not exist"
Nov 02 18:46:48.119144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:48.094047    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"]
Nov 02 18:46:48.119144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:48.098284    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134555-6b6c8c5499-8qlw6"]
Nov 02 18:46:48.202506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:48.202471    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"]
Nov 02 18:46:48.205235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:48.205207    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4fe24d/test-model-4fe24d-kserve-5d6855457c-p28tq"]
Nov 02 18:46:49.870429 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:49.868074    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5945f88d-c0bb-46b3-a915-3f0f17b84217" path="/var/lib/kubelet/pods/5945f88d-c0bb-46b3-a915-3f0f17b84217/volumes"
Nov 02 18:46:49.870429 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:46:49.868803    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31062a2-2f20-4e13-9133-9313d8f00f30" path="/var/lib/kubelet/pods/b31062a2-2f20-4e13-9133-9313d8f00f30/volumes"
Nov 02 18:46:59.055110 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:46:59.055068    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:47:00.274556 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.274519    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"]
Nov 02 18:47:00.275252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275231    4956 cpu_manager.go:414] "containerMap: removing container" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" containerName="test-curl-7517-1800eb"
Nov 02 18:47:00.275252 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275252    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" containerName="test-curl-7517-1800eb"
Nov 02 18:47:00.275391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275266    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b31062a2-2f20-4e13-9133-9313d8f00f30" containerName="main"
Nov 02 18:47:00.275391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275272    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31062a2-2f20-4e13-9133-9313d8f00f30" containerName="main"
Nov 02 18:47:00.275391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275287    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5945f88d-c0bb-46b3-a915-3f0f17b84217" containerName="payload-pre-processing"
Nov 02 18:47:00.275391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275292    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5945f88d-c0bb-46b3-a915-3f0f17b84217" containerName="payload-pre-processing"
Nov 02 18:47:00.275539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275413    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="072b9033-e88a-4b53-a72a-fbbd1a30d782" containerName="test-curl-7517-1800eb"
Nov 02 18:47:00.275539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275422    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31062a2-2f20-4e13-9133-9313d8f00f30" containerName="main"
Nov 02 18:47:00.275539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.275430    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5945f88d-c0bb-46b3-a915-3f0f17b84217" containerName="payload-pre-processing"
Nov 02 18:47:00.280495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.280473    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.332308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.332272    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c82472-openshift-default-dockercfg-kr6zx\""
Nov 02 18:47:00.376297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376250    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-token\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376487 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376379    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-data\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376487 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376461    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-certs\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376616 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376581    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istiod-ca-cert\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376675 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376623    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376675 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376651    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-credential-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376789 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376750    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-podinfo\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376782    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-envoy\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.376851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.376829    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnd66\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-kube-api-access-qnd66\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.385482 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.385448    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"]
Nov 02 18:47:00.478307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478269    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-envoy\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478320    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnd66\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-kube-api-access-qnd66\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478413    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-token\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478537 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478449    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-data\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478588    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-certs\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478664    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istiod-ca-cert\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478697    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478861 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478735    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-credential-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478861 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478808    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-podinfo\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.478861 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.478817    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-data\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.479030 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.479005    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.479091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.479058    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-certs\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.479411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.479386    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istiod-ca-cert\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.479534 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.479418    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-credential-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.480960 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.480937    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-envoy\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.481431 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.481411    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-podinfo\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.535626 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.535537    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-token\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.535794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.535654    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnd66\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-kube-api-access-qnd66\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:00.596831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:00.596794    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:01.026291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:01.026249    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"]
Nov 02 18:47:01.066484 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:47:01.066455    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1b89fc_03a4_4f67_bbef_e784a4fdb165.slice/crio-cd310d209a7c7d8d99a1ec56b2eecbe16b3c03ffa4db533906faffd98d7d1afa WatchSource:0}: Error finding container cd310d209a7c7d8d99a1ec56b2eecbe16b3c03ffa4db533906faffd98d7d1afa: Status 404 returned error can't find the container with id cd310d209a7c7d8d99a1ec56b2eecbe16b3c03ffa4db533906faffd98d7d1afa
Nov 02 18:47:01.069138 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:01.069088    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:47:01.069218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:01.069186    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:47:01.069256 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:01.069219    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:47:02.048146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:02.025720    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" event={"ID":"8b1b89fc-03a4-4f67-bbef-e784a4fdb165","Type":"ContainerStarted","Data":"85a4619b077460421cc47d41b287b283af939821789a32109888026bdf3ed294"}
Nov 02 18:47:02.048146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:02.025792    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" event={"ID":"8b1b89fc-03a4-4f67-bbef-e784a4fdb165","Type":"ContainerStarted","Data":"cd310d209a7c7d8d99a1ec56b2eecbe16b3c03ffa4db533906faffd98d7d1afa"}
Nov 02 18:47:02.260668 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:02.260601    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" podStartSLOduration=2.260583349 podStartE2EDuration="2.260583349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:02.208083076 +0000 UTC m=+5052.465113697" watchObservedRunningTime="2026-11-02 18:47:02.260583349 +0000 UTC m=+5052.517613968"
Nov 02 18:47:02.597875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:02.597836    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:02.599859 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:02.599832    4956 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:47:02.599991 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:02.599885    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 18:47:03.597851 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:03.597812    4956 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:47:03.598304 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:03.597868    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 18:47:04.597602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:04.597562    4956 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:47:04.597840 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:04.597642    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 18:47:05.598260 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:05.598221    4956 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:47:05.598707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:05.598279    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 18:47:06.597875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:06.597839    4956 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:47:06.598088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:06.597907    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 18:47:07.803945 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:07.803900    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:07.804631 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:07.804235    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:07.805677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:07.805637    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:47:29.053617 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:47:29.053577    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:47:29.155593 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.155549    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"]
Nov 02 18:47:29.155804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.155762    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7" podUID="5c125e9c-b65c-40bd-8313-c3f6c6b87de1" containerName="maas-api" containerID="cri-o://ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b" gracePeriod=30
Nov 02 18:47:29.432539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.432513    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:47:29.556677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.556581    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-openshift-service-ca\") pod \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") "
Nov 02 18:47:29.556677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.556645    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-metrics-certs\") pod \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") "
Nov 02 18:47:29.556930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.556808    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls\") pod \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") "
Nov 02 18:47:29.556930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.556916    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-kube-api-access-9gphr\" (UniqueName: \"kubernetes.io/projected/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-kube-api-access-9gphr\") pod \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\" (UID: \"5c125e9c-b65c-40bd-8313-c3f6c6b87de1\") "
Nov 02 18:47:29.561796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.561755    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-openshift-service-ca" pod "5c125e9c-b65c-40bd-8313-c3f6c6b87de1" (UID: "5c125e9c-b65c-40bd-8313-c3f6c6b87de1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:47:29.569027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.568988    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-kube-api-access-9gphr" pod "5c125e9c-b65c-40bd-8313-c3f6c6b87de1" (UID: "5c125e9c-b65c-40bd-8313-c3f6c6b87de1"). InnerVolumeSpecName "kube-api-access-9gphr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:29.572655 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.572621    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-metrics-certs" pod "5c125e9c-b65c-40bd-8313-c3f6c6b87de1" (UID: "5c125e9c-b65c-40bd-8313-c3f6c6b87de1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:29.581636 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.581605    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls" pod "5c125e9c-b65c-40bd-8313-c3f6c6b87de1" (UID: "5c125e9c-b65c-40bd-8313-c3f6c6b87de1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:29.659847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.659803    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gphr\" (UniqueName: \"kubernetes.io/projected/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-kube-api-access-9gphr\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:29.659847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.659842    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:29.659847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.659856    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:29.660106 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:29.659869    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c125e9c-b65c-40bd-8313-c3f6c6b87de1-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:30.228114 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.228071    4956 generic.go:356] "Generic (PLEG): container finished" podID="5c125e9c-b65c-40bd-8313-c3f6c6b87de1" containerID="ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b" exitCode=0
Nov 02 18:47:30.228650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.228161    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"
Nov 02 18:47:30.228650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.228155    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7" event={"ID":"5c125e9c-b65c-40bd-8313-c3f6c6b87de1","Type":"ContainerDied","Data":"ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b"}
Nov 02 18:47:30.228650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.228336    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7" event={"ID":"5c125e9c-b65c-40bd-8313-c3f6c6b87de1","Type":"ContainerDied","Data":"1e049d7d88e02c0a2d6882f4ac717fea0c5d2e2df68feb3ab0fc4afc9b485dbb"}
Nov 02 18:47:30.228650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.228369    4956 scope.go:122] "RemoveContainer" containerID="ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b"
Nov 02 18:47:30.240507 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.240487    4956 scope.go:122] "RemoveContainer" containerID="ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b"
Nov 02 18:47:30.240806 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:47:30.240784    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b\": container with ID starting with ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b not found: ID does not exist" containerID="ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b"
Nov 02 18:47:30.240870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.240816    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b"} err="failed to get container status \"ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b\": rpc error: code = NotFound desc = could not find container \"ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b\": container with ID starting with ec0bf49a1caa2df4e028f260214dbdb810a122ae8ba7ddb3d76103fe889dda4b not found: ID does not exist"
Nov 02 18:47:30.397036 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.396981    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"]
Nov 02 18:47:30.446470 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:30.446430    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134555-79567cb548-87sz7"]
Nov 02 18:47:31.866552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:31.866521    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c125e9c-b65c-40bd-8313-c3f6c6b87de1" path="/var/lib/kubelet/pods/5c125e9c-b65c-40bd-8313-c3f6c6b87de1/volumes"
Nov 02 18:47:50.619921 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:50.619888    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:47:50.628804 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:47:50.628776    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:47:59.053660 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:47:59.053622    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:48:19.669657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:19.669623    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"]
Nov 02 18:48:20.747197 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:20.747160    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"]
Nov 02 18:48:24.670564 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:24.670501    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" podUID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" containerName="payload-processing" containerID="cri-o://ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd" gracePeriod=25
Nov 02 18:48:29.052784 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:48:29.052750    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:48:30.739585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:30.739546    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" podUID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:48:35.768834 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:35.768772    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz" podUID="4d564e0b-5e57-4905-8209-548633dd9925" containerName="main" containerID="cri-o://84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f" gracePeriod=45
Nov 02 18:48:36.031576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.031497    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:48:36.055613 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055585    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-home\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-home\") pod \"4d564e0b-5e57-4905-8209-548633dd9925\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") "
Nov 02 18:48:36.055798 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055633    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d564e0b-5e57-4905-8209-548633dd9925-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d564e0b-5e57-4905-8209-548633dd9925-tls-certs\") pod \"4d564e0b-5e57-4905-8209-548633dd9925\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") "
Nov 02 18:48:36.055798 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055662    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-model-cache\") pod \"4d564e0b-5e57-4905-8209-548633dd9925\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") "
Nov 02 18:48:36.055798 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055688    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-dshm\") pod \"4d564e0b-5e57-4905-8209-548633dd9925\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") "
Nov 02 18:48:36.055971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055878    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-tmp-dir\") pod \"4d564e0b-5e57-4905-8209-548633dd9925\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") "
Nov 02 18:48:36.055971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055903    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-home" pod "4d564e0b-5e57-4905-8209-548633dd9925" (UID: "4d564e0b-5e57-4905-8209-548633dd9925"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:36.055971 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055951    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d564e0b-5e57-4905-8209-548633dd9925-kube-api-access-cts4m\" (UniqueName: \"kubernetes.io/projected/4d564e0b-5e57-4905-8209-548633dd9925-kube-api-access-cts4m\") pod \"4d564e0b-5e57-4905-8209-548633dd9925\" (UID: \"4d564e0b-5e57-4905-8209-548633dd9925\") "
Nov 02 18:48:36.056139 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.055949    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-model-cache" pod "4d564e0b-5e57-4905-8209-548633dd9925" (UID: "4d564e0b-5e57-4905-8209-548633dd9925"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:36.056187 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.056130    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-tmp-dir" pod "4d564e0b-5e57-4905-8209-548633dd9925" (UID: "4d564e0b-5e57-4905-8209-548633dd9925"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:36.056377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.056343    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:36.056377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.056366    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:36.056377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.056379    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:36.058104 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.058075    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-dshm" pod "4d564e0b-5e57-4905-8209-548633dd9925" (UID: "4d564e0b-5e57-4905-8209-548633dd9925"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:36.058218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.058145    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d564e0b-5e57-4905-8209-548633dd9925-tls-certs" pod "4d564e0b-5e57-4905-8209-548633dd9925" (UID: "4d564e0b-5e57-4905-8209-548633dd9925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:36.058218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.058205    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d564e0b-5e57-4905-8209-548633dd9925-kube-api-access-cts4m" pod "4d564e0b-5e57-4905-8209-548633dd9925" (UID: "4d564e0b-5e57-4905-8209-548633dd9925"). InnerVolumeSpecName "kube-api-access-cts4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:36.157685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.157654    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cts4m\" (UniqueName: \"kubernetes.io/projected/4d564e0b-5e57-4905-8209-548633dd9925-kube-api-access-cts4m\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:36.157685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.157683    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d564e0b-5e57-4905-8209-548633dd9925-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:36.157685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.157693    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d564e0b-5e57-4905-8209-548633dd9925-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:36.643951 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.643917    4956 generic.go:356] "Generic (PLEG): container finished" podID="4d564e0b-5e57-4905-8209-548633dd9925" containerID="84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f" exitCode=0
Nov 02 18:48:36.644153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.643958    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"
Nov 02 18:48:36.644153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.643968    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz" event={"ID":"4d564e0b-5e57-4905-8209-548633dd9925","Type":"ContainerDied","Data":"84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f"}
Nov 02 18:48:36.644153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.644007    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz" event={"ID":"4d564e0b-5e57-4905-8209-548633dd9925","Type":"ContainerDied","Data":"566b1644c1162efd7d4e4f7c1ecb2dc4d154f9cb667c39914d34014822d6160f"}
Nov 02 18:48:36.644153 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.644026    4956 scope.go:122] "RemoveContainer" containerID="84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f"
Nov 02 18:48:36.655386 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.655366    4956 scope.go:122] "RemoveContainer" containerID="84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f"
Nov 02 18:48:36.655689 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:48:36.655662    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f\": container with ID starting with 84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f not found: ID does not exist" containerID="84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f"
Nov 02 18:48:36.655748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.655702    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f"} err="failed to get container status \"84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f\": rpc error: code = NotFound desc = could not find container \"84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f\": container with ID starting with 84c82b028d3fb185a97f0f6d1892aa57462a1e2bf9cec8adbdecf7d01a5bb13f not found: ID does not exist"
Nov 02 18:48:36.806396 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.806357    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"]
Nov 02 18:48:36.855811 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:36.855769    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/sub-test-model-0ad26a-kserve-5bbf6ccdf-2vppz"]
Nov 02 18:48:37.865904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:37.865873    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d564e0b-5e57-4905-8209-548633dd9925" path="/var/lib/kubelet/pods/4d564e0b-5e57-4905-8209-548633dd9925/volumes"
Nov 02 18:48:40.739167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:40.739109    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" podUID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:48:44.940276 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:44.940217    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:48:45.061094 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.061064    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-kube-api-access-vmwf7\" (UniqueName: \"kubernetes.io/projected/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-kube-api-access-vmwf7\") pod \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") "
Nov 02 18:48:45.061324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.061248    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-plugins-config-volume\") pod \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\" (UID: \"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa\") "
Nov 02 18:48:45.061622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.061599    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-plugins-config-volume" pod "2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" (UID: "2125b82d-de3b-49e6-9ccf-3e1fcd785eaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:45.061724 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.061704    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.063445 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.063413    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-kube-api-access-vmwf7" pod "2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" (UID: "2125b82d-de3b-49e6-9ccf-3e1fcd785eaa"). InnerVolumeSpecName "kube-api-access-vmwf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:45.162950 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.162914    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmwf7\" (UniqueName: \"kubernetes.io/projected/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa-kube-api-access-vmwf7\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.732397 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.732346    4956 generic.go:356] "Generic (PLEG): container finished" podID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" containerID="ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd" exitCode=0
Nov 02 18:48:45.732618 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.732565    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" event={"ID":"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa","Type":"ContainerDied","Data":"ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd"}
Nov 02 18:48:45.732702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.732648    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p" event={"ID":"2125b82d-de3b-49e6-9ccf-3e1fcd785eaa","Type":"ContainerDied","Data":"e264f164162d90a77614cfeb93de5388f2ff1512506a596bb03dd0326881ce64"}
Nov 02 18:48:45.732702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.732679    4956 scope.go:122] "RemoveContainer" containerID="ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd"
Nov 02 18:48:45.732861 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.732844    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"
Nov 02 18:48:45.750561 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.750536    4956 scope.go:122] "RemoveContainer" containerID="ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd"
Nov 02 18:48:45.750855 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:48:45.750836    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd\": container with ID starting with ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd not found: ID does not exist" containerID="ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd"
Nov 02 18:48:45.750898 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.750866    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd"} err="failed to get container status \"ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd\": rpc error: code = NotFound desc = could not find container \"ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd\": container with ID starting with ce15f81545c2e63b62c7d8e523f7afcbb943999c8f00a6b8c1e8e0cb2e7b1fbd not found: ID does not exist"
Nov 02 18:48:45.896335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.896302    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"]
Nov 02 18:48:45.945970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:45.945941    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-413a4e-6964b6bd8f-bkl4p"]
Nov 02 18:48:47.865527 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:47.865490    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" path="/var/lib/kubelet/pods/2125b82d-de3b-49e6-9ccf-3e1fcd785eaa/volumes"
Nov 02 18:48:56.732419 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:56.732379    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"]
Nov 02 18:48:56.780330 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:48:56.780290    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"]
Nov 02 18:48:59.053109 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:48:59.053073    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:49:04.127886 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:04.127805    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"]
Nov 02 18:49:04.237488 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:04.237455    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"]
Nov 02 18:49:07.253944 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.253912    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"]
Nov 02 18:49:07.254405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.254166    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd" podUID="2fced4cc-74fe-4694-83a3-d3b2a80a8acb" containerName="maas-api" containerID="cri-o://0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c" gracePeriod=30
Nov 02 18:49:07.515820 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.515746    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:49:07.636773 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.636737    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls\") pod \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") "
Nov 02 18:49:07.636972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.636801    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-openshift-service-ca\") pod \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") "
Nov 02 18:49:07.636972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.636893    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-kube-api-access-lrzvd\" (UniqueName: \"kubernetes.io/projected/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-kube-api-access-lrzvd\") pod \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") "
Nov 02 18:49:07.637217 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.637163    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-metrics-certs\") pod \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\" (UID: \"2fced4cc-74fe-4694-83a3-d3b2a80a8acb\") "
Nov 02 18:49:07.637627 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.637595    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-openshift-service-ca" pod "2fced4cc-74fe-4694-83a3-d3b2a80a8acb" (UID: "2fced4cc-74fe-4694-83a3-d3b2a80a8acb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:07.637751 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.637737    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:07.640441 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.640410    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-kube-api-access-lrzvd" pod "2fced4cc-74fe-4694-83a3-d3b2a80a8acb" (UID: "2fced4cc-74fe-4694-83a3-d3b2a80a8acb"). InnerVolumeSpecName "kube-api-access-lrzvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:07.645897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.645867    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-metrics-certs" pod "2fced4cc-74fe-4694-83a3-d3b2a80a8acb" (UID: "2fced4cc-74fe-4694-83a3-d3b2a80a8acb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:07.646021 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.645972    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls" pod "2fced4cc-74fe-4694-83a3-d3b2a80a8acb" (UID: "2fced4cc-74fe-4694-83a3-d3b2a80a8acb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:07.738775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.738741    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:07.738775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.738771    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:07.738775 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.738783    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrzvd\" (UniqueName: \"kubernetes.io/projected/2fced4cc-74fe-4694-83a3-d3b2a80a8acb-kube-api-access-lrzvd\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:07.875845 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.875816    4956 generic.go:356] "Generic (PLEG): container finished" podID="2fced4cc-74fe-4694-83a3-d3b2a80a8acb" containerID="0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c" exitCode=0
Nov 02 18:49:07.875975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.875868    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"
Nov 02 18:49:07.875975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.875900    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd" event={"ID":"2fced4cc-74fe-4694-83a3-d3b2a80a8acb","Type":"ContainerDied","Data":"0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c"}
Nov 02 18:49:07.875975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.875947    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd" event={"ID":"2fced4cc-74fe-4694-83a3-d3b2a80a8acb","Type":"ContainerDied","Data":"0b65a06cb0e51ec1c01bc3b1b944265f8f67e9dd5c0fba021c46dff8fe548088"}
Nov 02 18:49:07.875975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.875968    4956 scope.go:122] "RemoveContainer" containerID="0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c"
Nov 02 18:49:07.886468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.886450    4956 scope.go:122] "RemoveContainer" containerID="0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c"
Nov 02 18:49:07.886756 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:07.886734    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c\": container with ID starting with 0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c not found: ID does not exist" containerID="0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c"
Nov 02 18:49:07.886835 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:07.886765    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c"} err="failed to get container status \"0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c\": rpc error: code = NotFound desc = could not find container \"0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c\": container with ID starting with 0e078db060db1dbe01180ac3310f0794ce096c50847f54c62dda6f31aec3209c not found: ID does not exist"
Nov 02 18:49:08.040746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:08.040707    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"]
Nov 02 18:49:08.043557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:08.043534    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-413a4e-6948dcb467-lxfsd"]
Nov 02 18:49:09.128388 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:09.128298    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" podUID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" containerName="payload-processing" containerID="cri-o://c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb" gracePeriod=25
Nov 02 18:49:09.237846 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:09.237807    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" podUID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" containerName="payload-pre-processing" containerID="cri-o://71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595" gracePeriod=25
Nov 02 18:49:09.865991 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:09.865955    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fced4cc-74fe-4694-83a3-d3b2a80a8acb" path="/var/lib/kubelet/pods/2fced4cc-74fe-4694-83a3-d3b2a80a8acb/volumes"
Nov 02 18:49:11.754310 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:11.754267    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main" containerID="cri-o://da218d1952897fb3097644030ae910c86fcdf614a4f5cdbe88b50593049cb7b4" gracePeriod=45
Nov 02 18:49:11.771642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:11.771611    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused"
Nov 02 18:49:11.801521 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:11.801487    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerName="main" containerID="cri-o://f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f" gracePeriod=45
Nov 02 18:49:11.915346 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:11.915309    4956 generic.go:356] "Generic (PLEG): container finished" podID="3c796a0b-d39e-45fd-86ac-13805db52493" containerID="da218d1952897fb3097644030ae910c86fcdf614a4f5cdbe88b50593049cb7b4" exitCode=0
Nov 02 18:49:11.915495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:11.915378    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" event={"ID":"3c796a0b-d39e-45fd-86ac-13805db52493","Type":"ContainerDied","Data":"da218d1952897fb3097644030ae910c86fcdf614a4f5cdbe88b50593049cb7b4"}
Nov 02 18:49:12.085979 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.085837    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 18:49:12.103888 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.103865    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 18:49:12.193602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193574    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-dshm\") pod \"3c796a0b-d39e-45fd-86ac-13805db52493\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") "
Nov 02 18:49:12.193780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193611    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c796a0b-d39e-45fd-86ac-13805db52493-kube-api-access-gwdgx\" (UniqueName: \"kubernetes.io/projected/3c796a0b-d39e-45fd-86ac-13805db52493-kube-api-access-gwdgx\") pod \"3c796a0b-d39e-45fd-86ac-13805db52493\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") "
Nov 02 18:49:12.193780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193669    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-tmp-dir\") pod \"d251df15-1ad4-4856-a2e0-939e2cdc083f\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") "
Nov 02 18:49:12.193780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193737    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-home\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-home\") pod \"3c796a0b-d39e-45fd-86ac-13805db52493\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") "
Nov 02 18:49:12.193947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193833    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d251df15-1ad4-4856-a2e0-939e2cdc083f-kube-api-access-xzwnf\" (UniqueName: \"kubernetes.io/projected/d251df15-1ad4-4856-a2e0-939e2cdc083f-kube-api-access-xzwnf\") pod \"d251df15-1ad4-4856-a2e0-939e2cdc083f\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") "
Nov 02 18:49:12.193947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193890    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-model-cache\") pod \"d251df15-1ad4-4856-a2e0-939e2cdc083f\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") "
Nov 02 18:49:12.194052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193963    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-tmp-dir" pod "d251df15-1ad4-4856-a2e0-939e2cdc083f" (UID: "d251df15-1ad4-4856-a2e0-939e2cdc083f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.194052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193991    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-home" pod "3c796a0b-d39e-45fd-86ac-13805db52493" (UID: "3c796a0b-d39e-45fd-86ac-13805db52493"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.194608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.193997    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c796a0b-d39e-45fd-86ac-13805db52493-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c796a0b-d39e-45fd-86ac-13805db52493-tls-certs\") pod \"3c796a0b-d39e-45fd-86ac-13805db52493\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") "
Nov 02 18:49:12.194608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194187    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-model-cache" pod "d251df15-1ad4-4856-a2e0-939e2cdc083f" (UID: "d251df15-1ad4-4856-a2e0-939e2cdc083f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.194608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194354    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-tmp-dir\") pod \"3c796a0b-d39e-45fd-86ac-13805db52493\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") "
Nov 02 18:49:12.194608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194563    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-tmp-dir" pod "3c796a0b-d39e-45fd-86ac-13805db52493" (UID: "3c796a0b-d39e-45fd-86ac-13805db52493"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.194948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194720    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-home" pod "d251df15-1ad4-4856-a2e0-939e2cdc083f" (UID: "d251df15-1ad4-4856-a2e0-939e2cdc083f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.194948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194804    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-home\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-home\") pod \"d251df15-1ad4-4856-a2e0-939e2cdc083f\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") "
Nov 02 18:49:12.194948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194879    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d251df15-1ad4-4856-a2e0-939e2cdc083f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d251df15-1ad4-4856-a2e0-939e2cdc083f-tls-certs\") pod \"d251df15-1ad4-4856-a2e0-939e2cdc083f\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") "
Nov 02 18:49:12.195112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194956    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-dshm\") pod \"d251df15-1ad4-4856-a2e0-939e2cdc083f\" (UID: \"d251df15-1ad4-4856-a2e0-939e2cdc083f\") "
Nov 02 18:49:12.195112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.194992    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-model-cache\") pod \"3c796a0b-d39e-45fd-86ac-13805db52493\" (UID: \"3c796a0b-d39e-45fd-86ac-13805db52493\") "
Nov 02 18:49:12.195485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.195465    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.195635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.195619    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.195932 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.195919    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.196096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.196083    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.196268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.196251    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.196413 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.195693    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-model-cache" pod "3c796a0b-d39e-45fd-86ac-13805db52493" (UID: "3c796a0b-d39e-45fd-86ac-13805db52493"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.196674 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.196644    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c796a0b-d39e-45fd-86ac-13805db52493-tls-certs" pod "3c796a0b-d39e-45fd-86ac-13805db52493" (UID: "3c796a0b-d39e-45fd-86ac-13805db52493"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:12.196763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.196694    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-dshm" pod "3c796a0b-d39e-45fd-86ac-13805db52493" (UID: "3c796a0b-d39e-45fd-86ac-13805db52493"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.196763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.196722    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d251df15-1ad4-4856-a2e0-939e2cdc083f-kube-api-access-xzwnf" pod "d251df15-1ad4-4856-a2e0-939e2cdc083f" (UID: "d251df15-1ad4-4856-a2e0-939e2cdc083f"). InnerVolumeSpecName "kube-api-access-xzwnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:12.196845 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.196766    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c796a0b-d39e-45fd-86ac-13805db52493-kube-api-access-gwdgx" pod "3c796a0b-d39e-45fd-86ac-13805db52493" (UID: "3c796a0b-d39e-45fd-86ac-13805db52493"). InnerVolumeSpecName "kube-api-access-gwdgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:12.197349 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.197330    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-dshm" pod "d251df15-1ad4-4856-a2e0-939e2cdc083f" (UID: "d251df15-1ad4-4856-a2e0-939e2cdc083f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.197646 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.197628    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d251df15-1ad4-4856-a2e0-939e2cdc083f-tls-certs" pod "d251df15-1ad4-4856-a2e0-939e2cdc083f" (UID: "d251df15-1ad4-4856-a2e0-939e2cdc083f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:12.297919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.297880    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwnf\" (UniqueName: \"kubernetes.io/projected/d251df15-1ad4-4856-a2e0-939e2cdc083f-kube-api-access-xzwnf\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.297919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.297913    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c796a0b-d39e-45fd-86ac-13805db52493-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.298194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.297929    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d251df15-1ad4-4856-a2e0-939e2cdc083f-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.298194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.297939    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d251df15-1ad4-4856-a2e0-939e2cdc083f-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.298194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.297949    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.298194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.297957    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c796a0b-d39e-45fd-86ac-13805db52493-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.298194 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.297965    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwdgx\" (UniqueName: \"kubernetes.io/projected/3c796a0b-d39e-45fd-86ac-13805db52493-kube-api-access-gwdgx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.923074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.923039    4956 generic.go:356] "Generic (PLEG): container finished" podID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerID="f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f" exitCode=0
Nov 02 18:49:12.923621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.923097    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"
Nov 02 18:49:12.923621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.923169    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" event={"ID":"d251df15-1ad4-4856-a2e0-939e2cdc083f","Type":"ContainerDied","Data":"f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f"}
Nov 02 18:49:12.923621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.923218    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x" event={"ID":"d251df15-1ad4-4856-a2e0-939e2cdc083f","Type":"ContainerDied","Data":"ee1ce750f7cad1cbce083d89021e0edce328263f46151072ddc73b492d943e86"}
Nov 02 18:49:12.923621 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.923243    4956 scope.go:122] "RemoveContainer" containerID="f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f"
Nov 02 18:49:12.924858 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.924834    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll" event={"ID":"3c796a0b-d39e-45fd-86ac-13805db52493","Type":"ContainerDied","Data":"6f652284f3f3804705e46ed37c452ecc53a57f844d542106071ebe4d2b1b123e"}
Nov 02 18:49:12.924945 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.924926    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"
Nov 02 18:49:12.935351 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.935329    4956 scope.go:122] "RemoveContainer" containerID="f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f"
Nov 02 18:49:12.935621 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:12.935598    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f\": container with ID starting with f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f not found: ID does not exist" containerID="f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f"
Nov 02 18:49:12.935693 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.935634    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f"} err="failed to get container status \"f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f\": rpc error: code = NotFound desc = could not find container \"f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f\": container with ID starting with f4bcc8400cde7a06930fad68b4c3c207e93cb9cdfe0ac60da9ce8f7624f5480f not found: ID does not exist"
Nov 02 18:49:12.935693 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:12.935656    4956 scope.go:122] "RemoveContainer" containerID="da218d1952897fb3097644030ae910c86fcdf614a4f5cdbe88b50593049cb7b4"
Nov 02 18:49:13.086340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:13.086248    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"]
Nov 02 18:49:13.135745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:13.135704    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-w4-4ad655-kserve-7c685b8654-d7w5x"]
Nov 02 18:49:13.198220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:13.198182    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"]
Nov 02 18:49:13.200562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:13.200540    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/distinct-2-w4-4ad655-kserve-6f5d67f757-lb6ll"]
Nov 02 18:49:13.417154 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:13.417036    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" podUID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:13.868669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:13.868629    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" path="/var/lib/kubelet/pods/3c796a0b-d39e-45fd-86ac-13805db52493/volumes"
Nov 02 18:49:13.869072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:13.869055    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" path="/var/lib/kubelet/pods/d251df15-1ad4-4856-a2e0-939e2cdc083f/volumes"
Nov 02 18:49:14.423927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:14.423888    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" podUID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:23.278966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:23.278930    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"]
Nov 02 18:49:23.386677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:23.386626    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"]
Nov 02 18:49:23.417115 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:23.417067    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" podUID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:24.423919 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:24.423880    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" podUID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:28.280001 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:28.279943    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerName="payload-processing" containerID="cri-o://5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1" gracePeriod=25
Nov 02 18:49:28.387190 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:28.387142    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" podUID="146210f3-688e-430f-a109-49704da37205" containerName="payload-pre-processing" containerID="cri-o://196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545" gracePeriod=25
Nov 02 18:49:28.533081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:28.532996    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" podUID="146210f3-688e-430f-a109-49704da37205" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:28.533081 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:28.533013    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:29.052536 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:29.052498    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:49:29.411466 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.411442    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:49:29.497385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.495982    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-plugins-config-volume\") pod \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") "
Nov 02 18:49:29.497385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.496163    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-kube-api-access-lb5jz\" (UniqueName: \"kubernetes.io/projected/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-kube-api-access-lb5jz\") pod \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\" (UID: \"1e3b0683-9211-4e11-8a35-c2c2a80b1c25\") "
Nov 02 18:49:29.497385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.497330    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-plugins-config-volume" pod "1e3b0683-9211-4e11-8a35-c2c2a80b1c25" (UID: "1e3b0683-9211-4e11-8a35-c2c2a80b1c25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:29.500467 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.500427    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-kube-api-access-lb5jz" pod "1e3b0683-9211-4e11-8a35-c2c2a80b1c25" (UID: "1e3b0683-9211-4e11-8a35-c2c2a80b1c25"). InnerVolumeSpecName "kube-api-access-lb5jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:29.521939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.521909    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:49:29.597453 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.597369    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-plugins-config-volume\") pod \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") "
Nov 02 18:49:29.597453 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.597430    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-kube-api-access-hsgn7\" (UniqueName: \"kubernetes.io/projected/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-kube-api-access-hsgn7\") pod \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\" (UID: \"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc\") "
Nov 02 18:49:29.597758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.597736    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-plugins-config-volume" pod "8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" (UID: "8045263d-4fe6-40b7-b83d-31ecc6c5f7cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:29.597810 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.597792    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:29.597810 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.597804    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb5jz\" (UniqueName: \"kubernetes.io/projected/1e3b0683-9211-4e11-8a35-c2c2a80b1c25-kube-api-access-lb5jz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:29.599716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.599695    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-kube-api-access-hsgn7" pod "8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" (UID: "8045263d-4fe6-40b7-b83d-31ecc6c5f7cc"). InnerVolumeSpecName "kube-api-access-hsgn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:29.699112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.699077    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:29.699112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:29.699112    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsgn7\" (UniqueName: \"kubernetes.io/projected/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc-kube-api-access-hsgn7\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:30.026632 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.026582    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"]
Nov 02 18:49:30.026854 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.026792    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv" podUID="416af64b-0e3c-4d42-b76c-5465fa83b650" containerName="maas-api" containerID="cri-o://e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f" gracePeriod=30
Nov 02 18:49:30.033849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.033821    4956 generic.go:356] "Generic (PLEG): container finished" podID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" containerID="71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595" exitCode=0
Nov 02 18:49:30.033985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.033865    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"
Nov 02 18:49:30.033985 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.033863    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" event={"ID":"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc","Type":"ContainerDied","Data":"71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595"}
Nov 02 18:49:30.034135 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.033980    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq" event={"ID":"8045263d-4fe6-40b7-b83d-31ecc6c5f7cc","Type":"ContainerDied","Data":"0294bd11cbff9a52a082aa1b3fba33fd68829086b167107958440d10ad93e6e3"}
Nov 02 18:49:30.034135 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.034007    4956 scope.go:122] "RemoveContainer" containerID="71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595"
Nov 02 18:49:30.035532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.035511    4956 generic.go:356] "Generic (PLEG): container finished" podID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" containerID="c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb" exitCode=0
Nov 02 18:49:30.035660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.035545    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" event={"ID":"1e3b0683-9211-4e11-8a35-c2c2a80b1c25","Type":"ContainerDied","Data":"c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb"}
Nov 02 18:49:30.035660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.035603    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r" event={"ID":"1e3b0683-9211-4e11-8a35-c2c2a80b1c25","Type":"ContainerDied","Data":"1fe6f0573f87cf1afdd8f20ee5beef30e1f6d58319ac77e2eacaaa2a6020948f"}
Nov 02 18:49:30.035660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.035556    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"
Nov 02 18:49:30.099223 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.099200    4956 scope.go:122] "RemoveContainer" containerID="71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595"
Nov 02 18:49:30.099984 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:30.099823    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595\": container with ID starting with 71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595 not found: ID does not exist" containerID="71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595"
Nov 02 18:49:30.099984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.099886    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595"} err="failed to get container status \"71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595\": rpc error: code = NotFound desc = could not find container \"71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595\": container with ID starting with 71d8075c3b89f005f031174eb66f3b91850b030b071bede37f5eabd8cebd8595 not found: ID does not exist"
Nov 02 18:49:30.099984 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.099906    4956 scope.go:122] "RemoveContainer" containerID="c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb"
Nov 02 18:49:30.141418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.141376    4956 scope.go:122] "RemoveContainer" containerID="c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb"
Nov 02 18:49:30.141821 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:30.141785    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb\": container with ID starting with c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb not found: ID does not exist" containerID="c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb"
Nov 02 18:49:30.141970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.141833    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb"} err="failed to get container status \"c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb\": rpc error: code = NotFound desc = could not find container \"c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb\": container with ID starting with c48869d048c375644a09514ac5b0204f401ef6c18c4cf60f4c9acc3df61226cb not found: ID does not exist"
Nov 02 18:49:30.252495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.252463    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"]
Nov 02 18:49:30.254486 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.254462    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0ad26a-7496475499-xlc5r"]
Nov 02 18:49:30.312809 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.312782    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 18:49:30.361027 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.360993    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"]
Nov 02 18:49:30.363785 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.363761    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0ad26a-85898fdcff-7t6xq"]
Nov 02 18:49:30.405612 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.405530    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-metrics-certs\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-metrics-certs\") pod \"416af64b-0e3c-4d42-b76c-5465fa83b650\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") "
Nov 02 18:49:30.405758 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.405682    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls\") pod \"416af64b-0e3c-4d42-b76c-5465fa83b650\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") "
Nov 02 18:49:30.405833 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.405757    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416af64b-0e3c-4d42-b76c-5465fa83b650-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416af64b-0e3c-4d42-b76c-5465fa83b650-openshift-service-ca\") pod \"416af64b-0e3c-4d42-b76c-5465fa83b650\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") "
Nov 02 18:49:30.405894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.405842    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416af64b-0e3c-4d42-b76c-5465fa83b650-kube-api-access-6ffg4\" (UniqueName: \"kubernetes.io/projected/416af64b-0e3c-4d42-b76c-5465fa83b650-kube-api-access-6ffg4\") pod \"416af64b-0e3c-4d42-b76c-5465fa83b650\" (UID: \"416af64b-0e3c-4d42-b76c-5465fa83b650\") "
Nov 02 18:49:30.406181 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.406148    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416af64b-0e3c-4d42-b76c-5465fa83b650-openshift-service-ca" pod "416af64b-0e3c-4d42-b76c-5465fa83b650" (UID: "416af64b-0e3c-4d42-b76c-5465fa83b650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:30.406390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.406285    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416af64b-0e3c-4d42-b76c-5465fa83b650-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:30.409219 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.409186    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-metrics-certs" pod "416af64b-0e3c-4d42-b76c-5465fa83b650" (UID: "416af64b-0e3c-4d42-b76c-5465fa83b650"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:30.409320 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.409265    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416af64b-0e3c-4d42-b76c-5465fa83b650-kube-api-access-6ffg4" pod "416af64b-0e3c-4d42-b76c-5465fa83b650" (UID: "416af64b-0e3c-4d42-b76c-5465fa83b650"). InnerVolumeSpecName "kube-api-access-6ffg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:30.409630 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.409606    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls" pod "416af64b-0e3c-4d42-b76c-5465fa83b650" (UID: "416af64b-0e3c-4d42-b76c-5465fa83b650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:30.508136 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.508090    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffg4\" (UniqueName: \"kubernetes.io/projected/416af64b-0e3c-4d42-b76c-5465fa83b650-kube-api-access-6ffg4\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:30.508523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.508145    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:30.508523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:30.508163    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416af64b-0e3c-4d42-b76c-5465fa83b650-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:31.043275 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.043244    4956 generic.go:356] "Generic (PLEG): container finished" podID="416af64b-0e3c-4d42-b76c-5465fa83b650" containerID="e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f" exitCode=0
Nov 02 18:49:31.043454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.043294    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"
Nov 02 18:49:31.043454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.043317    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv" event={"ID":"416af64b-0e3c-4d42-b76c-5465fa83b650","Type":"ContainerDied","Data":"e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f"}
Nov 02 18:49:31.043454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.043369    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv" event={"ID":"416af64b-0e3c-4d42-b76c-5465fa83b650","Type":"ContainerDied","Data":"a0488939695098b680b2f77550d60d87efd03a5e920bd9c89f0a4d50e4e2c835"}
Nov 02 18:49:31.043454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.043395    4956 scope.go:122] "RemoveContainer" containerID="e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f"
Nov 02 18:49:31.053624 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.053600    4956 scope.go:122] "RemoveContainer" containerID="e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f"
Nov 02 18:49:31.053916 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:31.053889    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f\": container with ID starting with e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f not found: ID does not exist" containerID="e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f"
Nov 02 18:49:31.053994 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.053925    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f"} err="failed to get container status \"e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f\": rpc error: code = NotFound desc = could not find container \"e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f\": container with ID starting with e31c1850e4ed8c9954f06f1ab1a6df730a114c68dda907ca3359b91c367fb95f not found: ID does not exist"
Nov 02 18:49:31.206235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.206200    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"]
Nov 02 18:49:31.255635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.255602    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ad655-7444fb8cbc-mfsjv"]
Nov 02 18:49:31.865517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.865483    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" path="/var/lib/kubelet/pods/1e3b0683-9211-4e11-8a35-c2c2a80b1c25/volumes"
Nov 02 18:49:31.866100 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.866079    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416af64b-0e3c-4d42-b76c-5465fa83b650" path="/var/lib/kubelet/pods/416af64b-0e3c-4d42-b76c-5465fa83b650/volumes"
Nov 02 18:49:31.866549 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:31.866528    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" path="/var/lib/kubelet/pods/8045263d-4fe6-40b7-b83d-31ecc6c5f7cc/volumes"
Nov 02 18:49:38.530343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:38.530300    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" podUID="146210f3-688e-430f-a109-49704da37205" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:38.530343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:38.530332    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:49:38.735611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:38.735577    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"]
Nov 02 18:49:38.735801 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:38.735766    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" containerID="cri-o://5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e" gracePeriod=30
Nov 02 18:49:39.229542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:39.229500    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"]
Nov 02 18:49:39.238980 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:39.238949    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"]
Nov 02 18:49:39.293493 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:39.293449    4956 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:49:39.293672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:39.293512    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:49:44.008479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.008406    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 18:49:44.068182 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068114    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istiod-ca-cert\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068202    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-data\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068325    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-token\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-token\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068359    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-envoy\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068394    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-certs\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068458    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-kube-api-access-bhl5k\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-kube-api-access-bhl5k\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068458    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istiod-ca-cert" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:44.068575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068535    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-socket\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068536    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-data" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:44.068827 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068589    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-credential-socket\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068827 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068647    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-podinfo\") pod \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\" (UID: \"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5\") "
Nov 02 18:49:44.068827 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068751    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-certs" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:44.068827 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068762    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-socket" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:44.069021 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.068852    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-credential-socket" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:44.069215 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.069196    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.069314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.069218    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.069314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.069232    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.069314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.069245    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.069314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.069259    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.070815 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.070793    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-kube-api-access-bhl5k" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "kube-api-access-bhl5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:44.070957 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.070816    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-token" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:44.071161 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.071140    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-envoy" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:44.071263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.071245    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-podinfo" pod "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" (UID: "d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:49:44.125415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.125383    4956 generic.go:356] "Generic (PLEG): container finished" podID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerID="5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e" exitCode=0
Nov 02 18:49:44.125595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.125436    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"
Nov 02 18:49:44.125595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.125444    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" event={"ID":"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5","Type":"ContainerDied","Data":"5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e"}
Nov 02 18:49:44.125595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.125486    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc" event={"ID":"d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5","Type":"ContainerDied","Data":"718ef2920b015569ae92c46e18c622ec6e61fa5f65a144bd43ebcad94fa93ce1"}
Nov 02 18:49:44.125595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.125509    4956 scope.go:122] "RemoveContainer" containerID="5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e"
Nov 02 18:49:44.135692 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.135674    4956 scope.go:122] "RemoveContainer" containerID="5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e"
Nov 02 18:49:44.135996 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:44.135977    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e\": container with ID starting with 5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e not found: ID does not exist" containerID="5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e"
Nov 02 18:49:44.136068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.136006    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e"} err="failed to get container status \"5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e\": rpc error: code = NotFound desc = could not find container \"5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e\": container with ID starting with 5f780989445719a864c0abd3dae2603ba21ec441e912fd3100b3628bbcbeed6e not found: ID does not exist"
Nov 02 18:49:44.170630 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.170598    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.170630 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.170630    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.170630 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.170645    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhl5k\" (UniqueName: \"kubernetes.io/projected/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-kube-api-access-bhl5k\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.170863 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.170658    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:44.285794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.285706    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"]
Nov 02 18:49:44.287914 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:44.287892    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-27bfc"]
Nov 02 18:49:45.864590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:45.864555    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" path="/var/lib/kubelet/pods/d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5/volumes"
Nov 02 18:49:48.530228 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:49:48.530199    4956 logging.go:55] [core] [Channel #18546 SubChannel #18548]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.127:50052", ServerName: "10.132.0.127:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.127:50052: connect: connection refused"
Nov 02 18:49:48.563145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.563103    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:49:48.659378 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.659350    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:49:48.721380 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.721344    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b8eabf-69c6-46c2-994a-33c73b88601f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b8eabf-69c6-46c2-994a-33c73b88601f-plugins-config-volume\") pod \"25b8eabf-69c6-46c2-994a-33c73b88601f\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") "
Nov 02 18:49:48.721541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.721428    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b8eabf-69c6-46c2-994a-33c73b88601f-kube-api-access-zpjrb\" (UniqueName: \"kubernetes.io/projected/25b8eabf-69c6-46c2-994a-33c73b88601f-kube-api-access-zpjrb\") pod \"25b8eabf-69c6-46c2-994a-33c73b88601f\" (UID: \"25b8eabf-69c6-46c2-994a-33c73b88601f\") "
Nov 02 18:49:48.721742 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.721717    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b8eabf-69c6-46c2-994a-33c73b88601f-plugins-config-volume" pod "25b8eabf-69c6-46c2-994a-33c73b88601f" (UID: "25b8eabf-69c6-46c2-994a-33c73b88601f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:48.723796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.723775    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b8eabf-69c6-46c2-994a-33c73b88601f-kube-api-access-zpjrb" pod "25b8eabf-69c6-46c2-994a-33c73b88601f" (UID: "25b8eabf-69c6-46c2-994a-33c73b88601f"). InnerVolumeSpecName "kube-api-access-zpjrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:48.822822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.822734    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146210f3-688e-430f-a109-49704da37205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146210f3-688e-430f-a109-49704da37205-plugins-config-volume\") pod \"146210f3-688e-430f-a109-49704da37205\" (UID: \"146210f3-688e-430f-a109-49704da37205\") "
Nov 02 18:49:48.822822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.822804    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146210f3-688e-430f-a109-49704da37205-kube-api-access-6zcvb\" (UniqueName: \"kubernetes.io/projected/146210f3-688e-430f-a109-49704da37205-kube-api-access-6zcvb\") pod \"146210f3-688e-430f-a109-49704da37205\" (UID: \"146210f3-688e-430f-a109-49704da37205\") "
Nov 02 18:49:48.823198 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.823171    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146210f3-688e-430f-a109-49704da37205-plugins-config-volume" pod "146210f3-688e-430f-a109-49704da37205" (UID: "146210f3-688e-430f-a109-49704da37205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:48.823405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.823388    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b8eabf-69c6-46c2-994a-33c73b88601f-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:48.823405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.823406    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjrb\" (UniqueName: \"kubernetes.io/projected/25b8eabf-69c6-46c2-994a-33c73b88601f-kube-api-access-zpjrb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:48.823712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.823414    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146210f3-688e-430f-a109-49704da37205-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:48.825061 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.825034    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146210f3-688e-430f-a109-49704da37205-kube-api-access-6zcvb" pod "146210f3-688e-430f-a109-49704da37205" (UID: "146210f3-688e-430f-a109-49704da37205"). InnerVolumeSpecName "kube-api-access-6zcvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:48.924620 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:48.924583    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zcvb\" (UniqueName: \"kubernetes.io/projected/146210f3-688e-430f-a109-49704da37205-kube-api-access-6zcvb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:49.156566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.156473    4956 generic.go:356] "Generic (PLEG): container finished" podID="146210f3-688e-430f-a109-49704da37205" containerID="196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545" exitCode=0
Nov 02 18:49:49.156566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.156517    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"
Nov 02 18:49:49.156566 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.156552    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" event={"ID":"146210f3-688e-430f-a109-49704da37205","Type":"ContainerDied","Data":"196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545"}
Nov 02 18:49:49.156860 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.156599    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" event={"ID":"146210f3-688e-430f-a109-49704da37205","Type":"ContainerDied","Data":"b7bb67f4518339bb34fb5f5dfd922dd5f1decd46e8c15af91986f6d7aee89e44"}
Nov 02 18:49:49.156860 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.156620    4956 scope.go:122] "RemoveContainer" containerID="196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545"
Nov 02 18:49:49.158390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.158099    4956 generic.go:356] "Generic (PLEG): container finished" podID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerID="5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1" exitCode=0
Nov 02 18:49:49.158390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.158162    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" event={"ID":"25b8eabf-69c6-46c2-994a-33c73b88601f","Type":"ContainerDied","Data":"5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1"}
Nov 02 18:49:49.158390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.158174    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"
Nov 02 18:49:49.158390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.158194    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" event={"ID":"25b8eabf-69c6-46c2-994a-33c73b88601f","Type":"ContainerDied","Data":"60308aad79ad48cd385a6523e6cd1c11140f4a1b0766c25a0abfca6e8db9c2c1"}
Nov 02 18:49:49.167542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.167512    4956 scope.go:122] "RemoveContainer" containerID="196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545"
Nov 02 18:49:49.167873 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:49.167845    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545\": container with ID starting with 196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545 not found: ID does not exist" containerID="196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545"
Nov 02 18:49:49.167935 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.167887    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545"} err="failed to get container status \"196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545\": rpc error: code = NotFound desc = could not find container \"196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545\": container with ID starting with 196bbbd3668d63aabd5a78c8af5c2929a17b2cefd393a5177ba0056e96aac545 not found: ID does not exist"
Nov 02 18:49:49.167935 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.167913    4956 scope.go:122] "RemoveContainer" containerID="5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1"
Nov 02 18:49:49.177094 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.177077    4956 scope.go:122] "RemoveContainer" containerID="5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1"
Nov 02 18:49:49.177430 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:49.177412    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1\": container with ID starting with 5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1 not found: ID does not exist" containerID="5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1"
Nov 02 18:49:49.177492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.177437    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1"} err="failed to get container status \"5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1\": rpc error: code = NotFound desc = could not find container \"5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1\": container with ID starting with 5d103cadc5a46e75c9943da6dc2cb80708e405e2fefee6aa7c8ac641dffd2da1 not found: ID does not exist"
Nov 02 18:49:49.318564 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.318521    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"]
Nov 02 18:49:49.322623 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.322596    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw"]
Nov 02 18:49:49.430177 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.430067    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"]
Nov 02 18:49:49.433688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.433658    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh"]
Nov 02 18:49:49.530212 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.530167    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c82472-d4d6fb5f-t96fw" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.128:50052\" within 1s: context deadline exceeded"
Nov 02 18:49:49.530393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.530179    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c82472-5446b87999-lb9kh" podUID="146210f3-688e-430f-a109-49704da37205" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.127:50052\" within 1s: context deadline exceeded"
Nov 02 18:49:49.530393 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:49:49.530287    4956 logging.go:55] [core] [Channel #18547 SubChannel #18549]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.128:50052", ServerName: "10.132.0.128:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.128:50052: operation was canceled"
Nov 02 18:49:49.865896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.865852    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146210f3-688e-430f-a109-49704da37205" path="/var/lib/kubelet/pods/146210f3-688e-430f-a109-49704da37205/volumes"
Nov 02 18:49:49.866261 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:49.866248    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" path="/var/lib/kubelet/pods/25b8eabf-69c6-46c2-994a-33c73b88601f/volumes"
Nov 02 18:49:54.252670 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.252604    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main" containerID="cri-o://37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb" gracePeriod=45
Nov 02 18:49:54.259073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.259013    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerName="main" containerID="cri-o://1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d" gracePeriod=45
Nov 02 18:49:54.517672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.517638    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 18:49:54.576385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.576360    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 18:49:54.689481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689446    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-kube-api-access-2n4th\" (UniqueName: \"kubernetes.io/projected/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-kube-api-access-2n4th\") pod \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") "
Nov 02 18:49:54.689667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689542    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tls-certs\") pod \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") "
Nov 02 18:49:54.689667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689594    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-dshm\") pod \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") "
Nov 02 18:49:54.689667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689633    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-dshm\") pod \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") "
Nov 02 18:49:54.689847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689749    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-kube-api-access-qkbgc\" (UniqueName: \"kubernetes.io/projected/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-kube-api-access-qkbgc\") pod \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") "
Nov 02 18:49:54.689847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689780    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tmp-dir\") pod \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") "
Nov 02 18:49:54.689847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689817    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tmp-dir\") pod \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") "
Nov 02 18:49:54.690018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689880    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tls-certs\") pod \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") "
Nov 02 18:49:54.690018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689930    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-home\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-home\") pod \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") "
Nov 02 18:49:54.690018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.689971    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-model-cache\") pod \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") "
Nov 02 18:49:54.690018 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.690011    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-model-cache\") pod \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\" (UID: \"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a\") "
Nov 02 18:49:54.690276 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.690075    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-home\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-home\") pod \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\" (UID: \"6e101ba8-4c2a-4657-ba2d-2164a2bd890d\") "
Nov 02 18:49:54.691481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.690792    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-home" pod "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" (UID: "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.691481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.690800    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-home" pod "6e101ba8-4c2a-4657-ba2d-2164a2bd890d" (UID: "6e101ba8-4c2a-4657-ba2d-2164a2bd890d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.691481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.690977    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-model-cache" pod "6e101ba8-4c2a-4657-ba2d-2164a2bd890d" (UID: "6e101ba8-4c2a-4657-ba2d-2164a2bd890d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.691481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.690983    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-model-cache" pod "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" (UID: "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.691481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.691433    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tmp-dir" pod "6e101ba8-4c2a-4657-ba2d-2164a2bd890d" (UID: "6e101ba8-4c2a-4657-ba2d-2164a2bd890d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.691481 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.691445    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tmp-dir" pod "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" (UID: "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.692732 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.692702    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tls-certs" pod "6e101ba8-4c2a-4657-ba2d-2164a2bd890d" (UID: "6e101ba8-4c2a-4657-ba2d-2164a2bd890d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:54.694469 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.694411    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-kube-api-access-qkbgc" pod "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" (UID: "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a"). InnerVolumeSpecName "kube-api-access-qkbgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:54.694469 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.694418    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-dshm" pod "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" (UID: "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.694622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.694481    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-kube-api-access-2n4th" pod "6e101ba8-4c2a-4657-ba2d-2164a2bd890d" (UID: "6e101ba8-4c2a-4657-ba2d-2164a2bd890d"). InnerVolumeSpecName "kube-api-access-2n4th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:54.694622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.694496    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tls-certs" pod "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" (UID: "fcdec34f-6eaf-44f6-920e-1deaa0af0c4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:54.696166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.696136    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-dshm" pod "6e101ba8-4c2a-4657-ba2d-2164a2bd890d" (UID: "6e101ba8-4c2a-4657-ba2d-2164a2bd890d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.791948    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4th\" (UniqueName: \"kubernetes.io/projected/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-kube-api-access-2n4th\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.791975    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.791985    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.791994    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792002    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbgc\" (UniqueName: \"kubernetes.io/projected/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-kube-api-access-qkbgc\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792012    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792021    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792028    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792037    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792046    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792053    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.792454 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:54.792063    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e101ba8-4c2a-4657-ba2d-2164a2bd890d-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:55.214983 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.214947    4956 generic.go:356] "Generic (PLEG): container finished" podID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerID="37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb" exitCode=0
Nov 02 18:49:55.215225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.215002    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"
Nov 02 18:49:55.215225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.215035    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" event={"ID":"6e101ba8-4c2a-4657-ba2d-2164a2bd890d","Type":"ContainerDied","Data":"37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb"}
Nov 02 18:49:55.215225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.215080    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" event={"ID":"6e101ba8-4c2a-4657-ba2d-2164a2bd890d","Type":"ContainerDied","Data":"8a2df0e8bfd14117581b8267aa08e4e50b73f6cd48e639f0bf9eb7b01297f316"}
Nov 02 18:49:55.215225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.215109    4956 scope.go:122] "RemoveContainer" containerID="37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb"
Nov 02 18:49:55.216546 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.216524    4956 generic.go:356] "Generic (PLEG): container finished" podID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerID="1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d" exitCode=0
Nov 02 18:49:55.216660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.216579    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" event={"ID":"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a","Type":"ContainerDied","Data":"1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d"}
Nov 02 18:49:55.216660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.216602    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"
Nov 02 18:49:55.216660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.216631    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29" event={"ID":"fcdec34f-6eaf-44f6-920e-1deaa0af0c4a","Type":"ContainerDied","Data":"70782368248aef025b3ba3b5da0f4325a17eba8de8f93351f5cdf20fd0c2a9e7"}
Nov 02 18:49:55.227225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.227014    4956 scope.go:122] "RemoveContainer" containerID="37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb"
Nov 02 18:49:55.227472 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:55.227443    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb\": container with ID starting with 37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb not found: ID does not exist" containerID="37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb"
Nov 02 18:49:55.227612 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.227485    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb"} err="failed to get container status \"37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb\": rpc error: code = NotFound desc = could not find container \"37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb\": container with ID starting with 37759de82faf8887a575662766f16ae4b5a2c58437d490895cb216cb7ddb3feb not found: ID does not exist"
Nov 02 18:49:55.227612 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.227508    4956 scope.go:122] "RemoveContainer" containerID="1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d"
Nov 02 18:49:55.237100 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.237081    4956 scope.go:122] "RemoveContainer" containerID="1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d"
Nov 02 18:49:55.237380 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:55.237361    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d\": container with ID starting with 1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d not found: ID does not exist" containerID="1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d"
Nov 02 18:49:55.237441 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.237389    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d"} err="failed to get container status \"1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d\": rpc error: code = NotFound desc = could not find container \"1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d\": container with ID starting with 1f8c7362943aaf1a19484e37228ed43e969efb4a2748dea7f8032df0c223396d not found: ID does not exist"
Nov 02 18:49:55.380826 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.380790    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"]
Nov 02 18:49:55.430469 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.430428    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/simulator-w1-dc5689-kserve-794866f44c-4xt29"]
Nov 02 18:49:55.509588 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.509457    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"]
Nov 02 18:49:55.512785 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.512733    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:49:55.513612 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.513585    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/premium-w1-dc5689-kserve-76968c6dbb-w9xhg"]
Nov 02 18:49:55.864610 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.864536    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" path="/var/lib/kubelet/pods/6e101ba8-4c2a-4657-ba2d-2164a2bd890d/volumes"
Nov 02 18:49:55.864991 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:49:55.864978    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" path="/var/lib/kubelet/pods/fcdec34f-6eaf-44f6-920e-1deaa0af0c4a/volumes"
Nov 02 18:49:59.052184 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:49:59.052152    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:50:00.967685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:00.967647    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"]
Nov 02 18:50:12.066816 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:12.066774    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"]
Nov 02 18:50:12.067292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:12.067019    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy" containerID="cri-o://85a4619b077460421cc47d41b287b283af939821789a32109888026bdf3ed294" gracePeriod=30
Nov 02 18:50:15.014312 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:15.014276    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"]
Nov 02 18:50:15.119894 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:15.119852    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"]
Nov 02 18:50:15.989327 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:15.989283    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main" containerID="cri-o://d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9" gracePeriod=45
Nov 02 18:50:16.267517 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.267488    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 18:50:16.343183 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.343153    4956 generic.go:356] "Generic (PLEG): container finished" podID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerID="d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9" exitCode=0
Nov 02 18:50:16.343390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.343212    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"
Nov 02 18:50:16.343390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.343252    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" event={"ID":"454bbc03-2858-4c60-a7ce-5e3b7079cec1","Type":"ContainerDied","Data":"d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9"}
Nov 02 18:50:16.343390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.343305    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" event={"ID":"454bbc03-2858-4c60-a7ce-5e3b7079cec1","Type":"ContainerDied","Data":"516ac45813368ccf1f8d75096586a07a82b12ebd6fb1eb13c4a959071b424685"}
Nov 02 18:50:16.343390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.343328    4956 scope.go:122] "RemoveContainer" containerID="d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9"
Nov 02 18:50:16.344203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344178    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-home\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-home\") pod \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") "
Nov 02 18:50:16.344203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344212    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tls-certs\" (UniqueName: \"kubernetes.io/secret/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tls-certs\") pod \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") "
Nov 02 18:50:16.344391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344247    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454bbc03-2858-4c60-a7ce-5e3b7079cec1-kube-api-access-hj49n\" (UniqueName: \"kubernetes.io/projected/454bbc03-2858-4c60-a7ce-5e3b7079cec1-kube-api-access-hj49n\") pod \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") "
Nov 02 18:50:16.344391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344300    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tmp-dir\") pod \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") "
Nov 02 18:50:16.344570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344426    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-model-cache\") pod \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") "
Nov 02 18:50:16.344570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344465    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-home" pod "454bbc03-2858-4c60-a7ce-5e3b7079cec1" (UID: "454bbc03-2858-4c60-a7ce-5e3b7079cec1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.344570 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344505    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-dshm\") pod \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\" (UID: \"454bbc03-2858-4c60-a7ce-5e3b7079cec1\") "
Nov 02 18:50:16.344738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344637    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tmp-dir" pod "454bbc03-2858-4c60-a7ce-5e3b7079cec1" (UID: "454bbc03-2858-4c60-a7ce-5e3b7079cec1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.344799 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344756    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-model-cache" pod "454bbc03-2858-4c60-a7ce-5e3b7079cec1" (UID: "454bbc03-2858-4c60-a7ce-5e3b7079cec1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.344961 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344941    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.345048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344966    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.345048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.344980    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.347014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.346986    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tls-certs" pod "454bbc03-2858-4c60-a7ce-5e3b7079cec1" (UID: "454bbc03-2858-4c60-a7ce-5e3b7079cec1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:16.347244 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.347213    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-dshm" pod "454bbc03-2858-4c60-a7ce-5e3b7079cec1" (UID: "454bbc03-2858-4c60-a7ce-5e3b7079cec1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.347341 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.347319    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454bbc03-2858-4c60-a7ce-5e3b7079cec1-kube-api-access-hj49n" pod "454bbc03-2858-4c60-a7ce-5e3b7079cec1" (UID: "454bbc03-2858-4c60-a7ce-5e3b7079cec1"). InnerVolumeSpecName "kube-api-access-hj49n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:16.369954 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.369933    4956 scope.go:122] "RemoveContainer" containerID="d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9"
Nov 02 18:50:16.370367 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:16.370338    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9\": container with ID starting with d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9 not found: ID does not exist" containerID="d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9"
Nov 02 18:50:16.370487 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.370380    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9"} err="failed to get container status \"d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9\": rpc error: code = NotFound desc = could not find container \"d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9\": container with ID starting with d1bcf222f695d4610267695f79d2a7e8d98687749100d69e433878781f3780c9 not found: ID does not exist"
Nov 02 18:50:16.445748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.445713    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/454bbc03-2858-4c60-a7ce-5e3b7079cec1-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.445748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.445739    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/454bbc03-2858-4c60-a7ce-5e3b7079cec1-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.445748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.445750    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj49n\" (UniqueName: \"kubernetes.io/projected/454bbc03-2858-4c60-a7ce-5e3b7079cec1-kube-api-access-hj49n\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.762709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.762673    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"]
Nov 02 18:50:16.767983 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:16.767948    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn"]
Nov 02 18:50:17.218492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.218441    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/simulator-w0-ceb1ae-kserve-7f79c8fb8b-sr4gn" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 18:50:17.354982 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.354945    4956 generic.go:356] "Generic (PLEG): container finished" podID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerID="85a4619b077460421cc47d41b287b283af939821789a32109888026bdf3ed294" exitCode=0
Nov 02 18:50:17.355412 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.355075    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" event={"ID":"8b1b89fc-03a4-4f67-bbef-e784a4fdb165","Type":"ContainerDied","Data":"85a4619b077460421cc47d41b287b283af939821789a32109888026bdf3ed294"}
Nov 02 18:50:17.381911 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.381885    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:50:17.457764 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.457729    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-token\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-token\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.457953 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.457873    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istiod-ca-cert\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.457953 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.457908    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-socket\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.457953 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.457936    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-envoy\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.458133 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.457973    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-podinfo\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.458133 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458013    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-certs\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.458133 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458070    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-credential-socket\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.458307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458201    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-socket" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:17.458307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458250    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-data\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.458307 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458293    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-kube-api-access-qnd66\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-kube-api-access-qnd66\") pod \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\" (UID: \"8b1b89fc-03a4-4f67-bbef-e784a4fdb165\") "
Nov 02 18:50:17.458480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458385    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-certs" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:17.458480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458412    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-credential-socket" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:17.458480 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458415    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istiod-ca-cert" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:17.458772 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458720    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.458772 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458744    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.458772 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458757    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.458772 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.458769    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.459237 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.459107    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-data" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:17.460695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.460670    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-token" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:17.460695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.460694    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-envoy" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:17.461024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.460996    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-podinfo" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:50:17.461112 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.461081    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-kube-api-access-qnd66" pod "8b1b89fc-03a4-4f67-bbef-e784a4fdb165" (UID: "8b1b89fc-03a4-4f67-bbef-e784a4fdb165"). InnerVolumeSpecName "kube-api-access-qnd66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:17.465149 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.465106    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"]
Nov 02 18:50:17.475882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.475853    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"]
Nov 02 18:50:17.560074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.560033    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.560074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.560060    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnd66\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-kube-api-access-qnd66\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.560074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.560071    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.560074 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.560080    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.560474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.560088    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1b89fc-03a4-4f67-bbef-e784a4fdb165-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:17.864322 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:17.864288    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" path="/var/lib/kubelet/pods/454bbc03-2858-4c60-a7ce-5e3b7079cec1/volumes"
Nov 02 18:50:18.363827 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:18.363799    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"
Nov 02 18:50:18.364300 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:18.363799    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw" event={"ID":"8b1b89fc-03a4-4f67-bbef-e784a4fdb165","Type":"ContainerDied","Data":"cd310d209a7c7d8d99a1ec56b2eecbe16b3c03ffa4db533906faffd98d7d1afa"}
Nov 02 18:50:18.364300 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:18.363931    4956 scope.go:122] "RemoveContainer" containerID="85a4619b077460421cc47d41b287b283af939821789a32109888026bdf3ed294"
Nov 02 18:50:18.527966 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:18.527928    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"]
Nov 02 18:50:18.577052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:18.577017    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-t64nw"]
Nov 02 18:50:19.864498 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:19.864460    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" path="/var/lib/kubelet/pods/8b1b89fc-03a4-4f67-bbef-e784a4fdb165/volumes"
Nov 02 18:50:20.015438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:20.015373    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" podUID="21f4460b-b73d-4949-827f-c59c14e5348d" containerName="payload-processing" containerID="cri-o://71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab" gracePeriod=25
Nov 02 18:50:20.120576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:20.120477    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" podUID="437e79f2-76bd-4221-a2f8-8bd890420692" containerName="payload-pre-processing" containerID="cri-o://a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f" gracePeriod=25
Nov 02 18:50:25.763598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.763558    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"]
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764075    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerName="main"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764087    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerName="main"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764097    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764103    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764130    4956 cpu_manager.go:414] "containerMap: removing container" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764135    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764142    4956 cpu_manager.go:414] "containerMap: removing container" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764147    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764154    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" containerName="payload-pre-processing"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764158    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" containerName="payload-pre-processing"
Nov 02 18:50:25.764159 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764166    4956 cpu_manager.go:414] "containerMap: removing container" podUID="2fced4cc-74fe-4694-83a3-d3b2a80a8acb" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764171    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fced4cc-74fe-4694-83a3-d3b2a80a8acb" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764180    4956 cpu_manager.go:414] "containerMap: removing container" podUID="146210f3-688e-430f-a109-49704da37205" containerName="payload-pre-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764185    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146210f3-688e-430f-a109-49704da37205" containerName="payload-pre-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764191    4956 cpu_manager.go:414] "containerMap: removing container" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764195    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764206    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764211    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764219    4956 cpu_manager.go:414] "containerMap: removing container" podUID="5c125e9c-b65c-40bd-8313-c3f6c6b87de1" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764223    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c125e9c-b65c-40bd-8313-c3f6c6b87de1" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764237    4956 cpu_manager.go:414] "containerMap: removing container" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764242    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764247    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4d564e0b-5e57-4905-8209-548633dd9925" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764253    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d564e0b-5e57-4905-8209-548633dd9925" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764267    4956 cpu_manager.go:414] "containerMap: removing container" podUID="416af64b-0e3c-4d42-b76c-5465fa83b650" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764271    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416af64b-0e3c-4d42-b76c-5465fa83b650" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764278    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764281    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764291    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764295    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764303    4956 cpu_manager.go:414] "containerMap: removing container" podUID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764307    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764374    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="454bbc03-2858-4c60-a7ce-5e3b7079cec1" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764383    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8045263d-4fe6-40b7-b83d-31ecc6c5f7cc" containerName="payload-pre-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764391    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fced4cc-74fe-4694-83a3-d3b2a80a8acb" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764397    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="146210f3-688e-430f-a109-49704da37205" containerName="payload-pre-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764402    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b8eabf-69c6-46c2-994a-33c73b88601f" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764407    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e101ba8-4c2a-4657-ba2d-2164a2bd890d" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764414    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c125e9c-b65c-40bd-8313-c3f6c6b87de1" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764422    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c796a0b-d39e-45fd-86ac-13805db52493" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764428    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d564e0b-5e57-4905-8209-548633dd9925" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764436    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="416af64b-0e3c-4d42-b76c-5465fa83b650" containerName="maas-api"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764441    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdec34f-6eaf-44f6-920e-1deaa0af0c4a" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764446    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3b0683-9211-4e11-8a35-c2c2a80b1c25" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764452    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="2125b82d-de3b-49e6-9ccf-3e1fcd785eaa" containerName="payload-processing"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764458    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d251df15-1ad4-4856-a2e0-939e2cdc083f" containerName="main"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764463    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1b89fc-03a4-4f67-bbef-e784a4fdb165" containerName="istio-proxy"
Nov 02 18:50:25.764576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.764469    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b939f8-20cf-4b13-8d25-b0ad0ba8c9f5" containerName="istio-proxy"
Nov 02 18:50:25.768943 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.768922    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.820605 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.820572    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\""
Nov 02 18:50:25.872788 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.872755    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"]
Nov 02 18:50:25.949796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.949739    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.949796 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.949789    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.950090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.949902    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.950090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.950063    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.950274 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.950114    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nxt\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-kube-api-access-z6nxt\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.950336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.950269    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.950336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.950295    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.950336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.950314    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:25.950534 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:25.950509    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052057    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052169    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052208    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052301    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052344    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052410    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nxt\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-kube-api-access-z6nxt\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052501    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052535    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052563    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052859 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052648    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052859 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052679    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.052859 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052742    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.053014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.052937    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.053167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.053144    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.055016 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.054993    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.055393 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.055370    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.109995 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.109955    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.110179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.110085    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nxt\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-kube-api-access-z6nxt\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.382556 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.382469    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:26.813508 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.813479    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"]
Nov 02 18:50:26.815081 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:50:26.815056    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e92823d_fd4e_40fa_a5a4_c4aab1ccdcca.slice/crio-062ed7a47315495e22d6df0de98fcfab04451b6ebf9cbfb91df66f794e301231 WatchSource:0}: Error finding container 062ed7a47315495e22d6df0de98fcfab04451b6ebf9cbfb91df66f794e301231: Status 404 returned error can't find the container with id 062ed7a47315495e22d6df0de98fcfab04451b6ebf9cbfb91df66f794e301231
Nov 02 18:50:26.817521 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.817478    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:50:26.817642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.817562    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:50:26.817642 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:26.817609    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:50:27.419632 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:27.419583    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" event={"ID":"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca","Type":"ContainerStarted","Data":"5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90"}
Nov 02 18:50:27.419824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:27.419654    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" event={"ID":"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca","Type":"ContainerStarted","Data":"062ed7a47315495e22d6df0de98fcfab04451b6ebf9cbfb91df66f794e301231"}
Nov 02 18:50:27.639146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:27.636453    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" podStartSLOduration=2.636434577 podStartE2EDuration="2.636434577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:27.583548322 +0000 UTC m=+5257.840578942" watchObservedRunningTime="2026-11-02 18:50:27.636434577 +0000 UTC m=+5257.893465197"
Nov 02 18:50:28.383014 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:28.382969    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:28.384596 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:28.384566    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:50:28.384716 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:28.384621    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 18:50:28.957474 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:28.957433    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"]
Nov 02 18:50:29.076853 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:29.076811    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:50:29.268171 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:29.268055    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" podUID="21f4460b-b73d-4949-827f-c59c14e5348d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:29.268171 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:29.268065    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" podUID="437e79f2-76bd-4221-a2f8-8bd890420692" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:29.383166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:29.383114    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:50:29.383622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:29.383192    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 18:50:29.455178 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:29.455144    4956 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" secret="" err="secret \"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\" not found"
Nov 02 18:50:29.789584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:29.789529    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"]
Nov 02 18:50:29.789838 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:29.789810    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r" podUID="8ce42215-7e35-4588-9083-54c2c9611934" containerName="maas-api" containerID="cri-o://84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec" gracePeriod=30
Nov 02 18:50:30.115382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.115355    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 18:50:30.198637 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.198606    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls\") pod \"8ce42215-7e35-4588-9083-54c2c9611934\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") "
Nov 02 18:50:30.198824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.198759    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce42215-7e35-4588-9083-54c2c9611934-kube-api-access-62hrp\" (UniqueName: \"kubernetes.io/projected/8ce42215-7e35-4588-9083-54c2c9611934-kube-api-access-62hrp\") pod \"8ce42215-7e35-4588-9083-54c2c9611934\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") "
Nov 02 18:50:30.198824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.198778    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce42215-7e35-4588-9083-54c2c9611934-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce42215-7e35-4588-9083-54c2c9611934-openshift-service-ca\") pod \"8ce42215-7e35-4588-9083-54c2c9611934\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") "
Nov 02 18:50:30.198824 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.198808    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-metrics-certs\") pod \"8ce42215-7e35-4588-9083-54c2c9611934\" (UID: \"8ce42215-7e35-4588-9083-54c2c9611934\") "
Nov 02 18:50:30.199999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.199142    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce42215-7e35-4588-9083-54c2c9611934-openshift-service-ca" pod "8ce42215-7e35-4588-9083-54c2c9611934" (UID: "8ce42215-7e35-4588-9083-54c2c9611934"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:30.199999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.199252    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce42215-7e35-4588-9083-54c2c9611934-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:30.200879 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.200846    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls" pod "8ce42215-7e35-4588-9083-54c2c9611934" (UID: "8ce42215-7e35-4588-9083-54c2c9611934"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:30.201251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.201230    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-metrics-certs" pod "8ce42215-7e35-4588-9083-54c2c9611934" (UID: "8ce42215-7e35-4588-9083-54c2c9611934"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:30.201329 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.201262    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce42215-7e35-4588-9083-54c2c9611934-kube-api-access-62hrp" pod "8ce42215-7e35-4588-9083-54c2c9611934" (UID: "8ce42215-7e35-4588-9083-54c2c9611934"). InnerVolumeSpecName "kube-api-access-62hrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:30.300323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.300282    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:30.300323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.300318    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62hrp\" (UniqueName: \"kubernetes.io/projected/8ce42215-7e35-4588-9083-54c2c9611934-kube-api-access-62hrp\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:30.300323 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.300330    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce42215-7e35-4588-9083-54c2c9611934-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:30.383359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.383264    4956 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:50:30.383359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.383329    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 18:50:30.462021 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.461984    4956 generic.go:356] "Generic (PLEG): container finished" podID="8ce42215-7e35-4588-9083-54c2c9611934" containerID="84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec" exitCode=0
Nov 02 18:50:30.462218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.462034    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"
Nov 02 18:50:30.462218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.462073    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r" event={"ID":"8ce42215-7e35-4588-9083-54c2c9611934","Type":"ContainerDied","Data":"84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec"}
Nov 02 18:50:30.462218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.462145    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r" event={"ID":"8ce42215-7e35-4588-9083-54c2c9611934","Type":"ContainerDied","Data":"aa865d3e2bed90381cfb1d4d5833cdd7d862d99dc161252fc8e3c3d0e2bb0b45"}
Nov 02 18:50:30.462218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.462169    4956 scope.go:122] "RemoveContainer" containerID="84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec"
Nov 02 18:50:30.462482 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.462452    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerName="istio-proxy" containerID="cri-o://5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90" gracePeriod=30
Nov 02 18:50:30.472855 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.472837    4956 scope.go:122] "RemoveContainer" containerID="84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec"
Nov 02 18:50:30.473170 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:30.473152    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec\": container with ID starting with 84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec not found: ID does not exist" containerID="84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec"
Nov 02 18:50:30.473247 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.473182    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec"} err="failed to get container status \"84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec\": rpc error: code = NotFound desc = could not find container \"84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec\": container with ID starting with 84298d6cc562a8ac26260540e57799e1258f84424684ffb7bc9d072edbbcaaec not found: ID does not exist"
Nov 02 18:50:30.640208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.627983    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"]
Nov 02 18:50:30.654324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:30.654284    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ceb1ae-74698fdd75-npz5r"]
Nov 02 18:50:31.154331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:31.154294    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"]
Nov 02 18:50:31.865407 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:31.865373    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce42215-7e35-4588-9083-54c2c9611934" path="/var/lib/kubelet/pods/8ce42215-7e35-4588-9083-54c2c9611934/volumes"
Nov 02 18:50:32.488065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.488026    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main" containerID="cri-o://42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729" gracePeriod=45
Nov 02 18:50:32.496288 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.496253    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main" containerID="cri-o://69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e" gracePeriod=45
Nov 02 18:50:32.558096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.558042    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused"
Nov 02 18:50:32.823657 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.823628    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 18:50:32.866607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.866577    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 18:50:32.934604 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934569    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tmp-dir\") pod \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") "
Nov 02 18:50:32.934821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934641    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-home\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-home\") pod \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") "
Nov 02 18:50:32.934821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934712    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6389d6f-249f-4998-8760-30c38316ba91-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6389d6f-249f-4998-8760-30c38316ba91-tls-certs\") pod \"b6389d6f-249f-4998-8760-30c38316ba91\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") "
Nov 02 18:50:32.934821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934752    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-model-cache\") pod \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") "
Nov 02 18:50:32.934821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934781    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-tmp-dir\") pod \"b6389d6f-249f-4998-8760-30c38316ba91\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") "
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934854    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-dshm\") pod \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") "
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934888    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-home\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-home\") pod \"b6389d6f-249f-4998-8760-30c38316ba91\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") "
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934921    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-model-cache\") pod \"b6389d6f-249f-4998-8760-30c38316ba91\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") "
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934956    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6de910d-3a7e-47f8-957f-3d2735c9bf27-kube-api-access-c2rgx\" (UniqueName: \"kubernetes.io/projected/b6de910d-3a7e-47f8-957f-3d2735c9bf27-kube-api-access-c2rgx\") pod \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") "
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934964    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tmp-dir" pod "b6de910d-3a7e-47f8-957f-3d2735c9bf27" (UID: "b6de910d-3a7e-47f8-957f-3d2735c9bf27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.934994    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tls-certs\") pod \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\" (UID: \"b6de910d-3a7e-47f8-957f-3d2735c9bf27\") "
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935006    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-home" pod "b6de910d-3a7e-47f8-957f-3d2735c9bf27" (UID: "b6de910d-3a7e-47f8-957f-3d2735c9bf27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:32.935060 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935039    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6389d6f-249f-4998-8760-30c38316ba91-kube-api-access-bqql6\" (UniqueName: \"kubernetes.io/projected/b6389d6f-249f-4998-8760-30c38316ba91-kube-api-access-bqql6\") pod \"b6389d6f-249f-4998-8760-30c38316ba91\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") "
Nov 02 18:50:32.935471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935092    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-dshm\") pod \"b6389d6f-249f-4998-8760-30c38316ba91\" (UID: \"b6389d6f-249f-4998-8760-30c38316ba91\") "
Nov 02 18:50:32.935471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935172    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-model-cache" pod "b6389d6f-249f-4998-8760-30c38316ba91" (UID: "b6389d6f-249f-4998-8760-30c38316ba91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:32.935471 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935316    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-model-cache" pod "b6de910d-3a7e-47f8-957f-3d2735c9bf27" (UID: "b6de910d-3a7e-47f8-957f-3d2735c9bf27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:32.935710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935536    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:32.935710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935557    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:32.935710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935571    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:32.935710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935584    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:32.935955 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935753    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-tmp-dir" pod "b6389d6f-249f-4998-8760-30c38316ba91" (UID: "b6389d6f-249f-4998-8760-30c38316ba91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:32.936033 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.935683    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-home" pod "b6389d6f-249f-4998-8760-30c38316ba91" (UID: "b6389d6f-249f-4998-8760-30c38316ba91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:32.938690 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.938638    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6de910d-3a7e-47f8-957f-3d2735c9bf27-kube-api-access-c2rgx" pod "b6de910d-3a7e-47f8-957f-3d2735c9bf27" (UID: "b6de910d-3a7e-47f8-957f-3d2735c9bf27"). InnerVolumeSpecName "kube-api-access-c2rgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:32.938847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.938732    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tls-certs" pod "b6de910d-3a7e-47f8-957f-3d2735c9bf27" (UID: "b6de910d-3a7e-47f8-957f-3d2735c9bf27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:32.938847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.938759    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-dshm" pod "b6389d6f-249f-4998-8760-30c38316ba91" (UID: "b6389d6f-249f-4998-8760-30c38316ba91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:32.938847 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.938772    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6389d6f-249f-4998-8760-30c38316ba91-kube-api-access-bqql6" pod "b6389d6f-249f-4998-8760-30c38316ba91" (UID: "b6389d6f-249f-4998-8760-30c38316ba91"). InnerVolumeSpecName "kube-api-access-bqql6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:32.940203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.940176    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6389d6f-249f-4998-8760-30c38316ba91-tls-certs" pod "b6389d6f-249f-4998-8760-30c38316ba91" (UID: "b6389d6f-249f-4998-8760-30c38316ba91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:32.940500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:32.940481    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-dshm" pod "b6de910d-3a7e-47f8-957f-3d2735c9bf27" (UID: "b6de910d-3a7e-47f8-957f-3d2735c9bf27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:33.036698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036595    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6de910d-3a7e-47f8-957f-3d2735c9bf27-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.036698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036634    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.036698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036649    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2rgx\" (UniqueName: \"kubernetes.io/projected/b6de910d-3a7e-47f8-957f-3d2735c9bf27-kube-api-access-c2rgx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.036698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036665    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6de910d-3a7e-47f8-957f-3d2735c9bf27-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.036698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036681    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqql6\" (UniqueName: \"kubernetes.io/projected/b6389d6f-249f-4998-8760-30c38316ba91-kube-api-access-bqql6\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.036698 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036694    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.037263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036708    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6389d6f-249f-4998-8760-30c38316ba91-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.037263 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.036721    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6389d6f-249f-4998-8760-30c38316ba91-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:33.490616 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.490585    4956 generic.go:356] "Generic (PLEG): container finished" podID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerID="69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e" exitCode=0
Nov 02 18:50:33.490823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.490645    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"
Nov 02 18:50:33.490823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.490678    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" event={"ID":"b6de910d-3a7e-47f8-957f-3d2735c9bf27","Type":"ContainerDied","Data":"69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e"}
Nov 02 18:50:33.490823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.490725    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b" event={"ID":"b6de910d-3a7e-47f8-957f-3d2735c9bf27","Type":"ContainerDied","Data":"5ee43ff21dc1d326ea57177bc61948d25d2aebecf97fcdcf9d4adccbc19fdb65"}
Nov 02 18:50:33.490823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.490742    4956 scope.go:122] "RemoveContainer" containerID="69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e"
Nov 02 18:50:33.492314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.492003    4956 generic.go:356] "Generic (PLEG): container finished" podID="b6389d6f-249f-4998-8760-30c38316ba91" containerID="42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729" exitCode=0
Nov 02 18:50:33.492314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.492149    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" event={"ID":"b6389d6f-249f-4998-8760-30c38316ba91","Type":"ContainerDied","Data":"42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729"}
Nov 02 18:50:33.492314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.492173    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" event={"ID":"b6389d6f-249f-4998-8760-30c38316ba91","Type":"ContainerDied","Data":"ef4292e3b104935c8f902a45dc332e94018c5c411c659cc83f89c0623468f7dd"}
Nov 02 18:50:33.492314 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.492221    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"
Nov 02 18:50:33.500687 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.500669    4956 scope.go:122] "RemoveContainer" containerID="69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e"
Nov 02 18:50:33.500992 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:33.500974    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e\": container with ID starting with 69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e not found: ID does not exist" containerID="69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e"
Nov 02 18:50:33.501069 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.500998    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e"} err="failed to get container status \"69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e\": rpc error: code = NotFound desc = could not find container \"69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e\": container with ID starting with 69041e69a8adad75301dd439b75452c8da9edb290490fb52e90a19763486143e not found: ID does not exist"
Nov 02 18:50:33.501069 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.501014    4956 scope.go:122] "RemoveContainer" containerID="42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729"
Nov 02 18:50:33.519056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.519035    4956 scope.go:122] "RemoveContainer" containerID="42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729"
Nov 02 18:50:33.519431 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:33.519407    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729\": container with ID starting with 42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729 not found: ID does not exist" containerID="42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729"
Nov 02 18:50:33.519491 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.519443    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729"} err="failed to get container status \"42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729\": rpc error: code = NotFound desc = could not find container \"42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729\": container with ID starting with 42ffceeb802953199783143320d3010f50546041924c816e0d225cc9b9524729 not found: ID does not exist"
Nov 02 18:50:33.660220 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.660180    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"]
Nov 02 18:50:33.708199 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.708157    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w"]
Nov 02 18:50:33.725196 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.725110    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/premium-w2-a099c3-kserve-7779d88cf5-rbj2w" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:50:33.768498 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.768403    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"]
Nov 02 18:50:33.772552 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.772514    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/simulator-w2-a099c3-kserve-6494cc8fdb-s2l9b"]
Nov 02 18:50:33.867338 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.867299    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6389d6f-249f-4998-8760-30c38316ba91" path="/var/lib/kubelet/pods/b6389d6f-249f-4998-8760-30c38316ba91/volumes"
Nov 02 18:50:33.867837 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:33.867747    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" path="/var/lib/kubelet/pods/b6de910d-3a7e-47f8-957f-3d2735c9bf27/volumes"
Nov 02 18:50:36.155617 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:36.155572    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" podUID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" containerName="payload-pre-processing" containerID="cri-o://09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963" gracePeriod=25
Nov 02 18:50:38.115717 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:38.115671    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" podUID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:39.133860 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.133829    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:39.204383 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204352    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-kube-api-access-z6nxt\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-kube-api-access-z6nxt\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204391    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-token\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-token\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204426    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-certs\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204493    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-envoy\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204562 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204517    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-data\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204568    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-credential-socket\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204658    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-socket\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204708    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istiod-ca-cert\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.204780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204752    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-podinfo\") pod \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\" (UID: \"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca\") "
Nov 02 18:50:39.205012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204765    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-certs" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:39.205012 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.204983    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-socket" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:39.205229 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.205176    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-credential-socket" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:39.205229 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.205201    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-data" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:39.205363 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.205343    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.205405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.205355    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istiod-ca-cert" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:39.205405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.205370    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.205405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.205385    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.205405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.205398    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.206843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.206815    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-token" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:39.207037 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.207019    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-podinfo" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:50:39.207409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.207382    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-envoy" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:39.207409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.207389    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-kube-api-access-z6nxt" pod "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" (UID: "8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca"). InnerVolumeSpecName "kube-api-access-z6nxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:39.268065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.268025    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" podUID="21f4460b-b73d-4949-827f-c59c14e5348d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:39.268065 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.268056    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" podUID="437e79f2-76bd-4221-a2f8-8bd890420692" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:39.306910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.306874    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.306910 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.306905    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.307138 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.306920    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.307138 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.306932    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6nxt\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-kube-api-access-z6nxt\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.307138 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.306945    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.530868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.530776    4956 generic.go:356] "Generic (PLEG): container finished" podID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerID="5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90" exitCode=0
Nov 02 18:50:39.530868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.530826    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"
Nov 02 18:50:39.531063 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.530871    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" event={"ID":"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca","Type":"ContainerDied","Data":"5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90"}
Nov 02 18:50:39.531063 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.530940    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n" event={"ID":"8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca","Type":"ContainerDied","Data":"062ed7a47315495e22d6df0de98fcfab04451b6ebf9cbfb91df66f794e301231"}
Nov 02 18:50:39.531063 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.530968    4956 scope.go:122] "RemoveContainer" containerID="5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90"
Nov 02 18:50:39.546835 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.546809    4956 scope.go:122] "RemoveContainer" containerID="5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90"
Nov 02 18:50:39.547206 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:39.547185    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90\": container with ID starting with 5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90 not found: ID does not exist" containerID="5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90"
Nov 02 18:50:39.547321 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.547214    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90"} err="failed to get container status \"5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90\": rpc error: code = NotFound desc = could not find container \"5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90\": container with ID starting with 5cecbb090c323ac8cad8244ebe4b3f2c62d1325ec477c04ed448673dd4abed90 not found: ID does not exist"
Nov 02 18:50:39.694673 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.694634    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"]
Nov 02 18:50:39.699547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.699517    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fhf7n"]
Nov 02 18:50:39.864174 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:39.864075    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" path="/var/lib/kubelet/pods/8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca/volumes"
Nov 02 18:50:40.284499 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.284470    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 18:50:40.394781 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.394750    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 18:50:40.417053 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.417011    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f4460b-b73d-4949-827f-c59c14e5348d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f4460b-b73d-4949-827f-c59c14e5348d-plugins-config-volume\") pod \"21f4460b-b73d-4949-827f-c59c14e5348d\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") "
Nov 02 18:50:40.417256 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.417110    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f4460b-b73d-4949-827f-c59c14e5348d-kube-api-access-msv4x\" (UniqueName: \"kubernetes.io/projected/21f4460b-b73d-4949-827f-c59c14e5348d-kube-api-access-msv4x\") pod \"21f4460b-b73d-4949-827f-c59c14e5348d\" (UID: \"21f4460b-b73d-4949-827f-c59c14e5348d\") "
Nov 02 18:50:40.417529 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.417500    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f4460b-b73d-4949-827f-c59c14e5348d-plugins-config-volume" pod "21f4460b-b73d-4949-827f-c59c14e5348d" (UID: "21f4460b-b73d-4949-827f-c59c14e5348d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:40.417610 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.417587    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f4460b-b73d-4949-827f-c59c14e5348d-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:40.419523 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.419498    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f4460b-b73d-4949-827f-c59c14e5348d-kube-api-access-msv4x" pod "21f4460b-b73d-4949-827f-c59c14e5348d" (UID: "21f4460b-b73d-4949-827f-c59c14e5348d"). InnerVolumeSpecName "kube-api-access-msv4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:40.518834 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.518799    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/437e79f2-76bd-4221-a2f8-8bd890420692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/437e79f2-76bd-4221-a2f8-8bd890420692-plugins-config-volume\") pod \"437e79f2-76bd-4221-a2f8-8bd890420692\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") "
Nov 02 18:50:40.519195 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.518902    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437e79f2-76bd-4221-a2f8-8bd890420692-kube-api-access-xvggj\" (UniqueName: \"kubernetes.io/projected/437e79f2-76bd-4221-a2f8-8bd890420692-kube-api-access-xvggj\") pod \"437e79f2-76bd-4221-a2f8-8bd890420692\" (UID: \"437e79f2-76bd-4221-a2f8-8bd890420692\") "
Nov 02 18:50:40.519301 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.519265    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437e79f2-76bd-4221-a2f8-8bd890420692-plugins-config-volume" pod "437e79f2-76bd-4221-a2f8-8bd890420692" (UID: "437e79f2-76bd-4221-a2f8-8bd890420692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:40.519409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.519391    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msv4x\" (UniqueName: \"kubernetes.io/projected/21f4460b-b73d-4949-827f-c59c14e5348d-kube-api-access-msv4x\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:40.519467 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.519413    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/437e79f2-76bd-4221-a2f8-8bd890420692-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:40.521453 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.521405    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437e79f2-76bd-4221-a2f8-8bd890420692-kube-api-access-xvggj" pod "437e79f2-76bd-4221-a2f8-8bd890420692" (UID: "437e79f2-76bd-4221-a2f8-8bd890420692"). InnerVolumeSpecName "kube-api-access-xvggj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:40.539088 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.539055    4956 generic.go:356] "Generic (PLEG): container finished" podID="21f4460b-b73d-4949-827f-c59c14e5348d" containerID="71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab" exitCode=0
Nov 02 18:50:40.539359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.539104    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"
Nov 02 18:50:40.539359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.539148    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" event={"ID":"21f4460b-b73d-4949-827f-c59c14e5348d","Type":"ContainerDied","Data":"71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab"}
Nov 02 18:50:40.539359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.539208    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc" event={"ID":"21f4460b-b73d-4949-827f-c59c14e5348d","Type":"ContainerDied","Data":"a254af3c15f77167d4b8f956c36ad9c482801248a653a6852b4d18c36c6c5224"}
Nov 02 18:50:40.539359 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.539239    4956 scope.go:122] "RemoveContainer" containerID="71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab"
Nov 02 18:50:40.541210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.541086    4956 generic.go:356] "Generic (PLEG): container finished" podID="437e79f2-76bd-4221-a2f8-8bd890420692" containerID="a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f" exitCode=0
Nov 02 18:50:40.541210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.541160    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"
Nov 02 18:50:40.541210 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.541182    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" event={"ID":"437e79f2-76bd-4221-a2f8-8bd890420692","Type":"ContainerDied","Data":"a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f"}
Nov 02 18:50:40.541449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.541257    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr" event={"ID":"437e79f2-76bd-4221-a2f8-8bd890420692","Type":"ContainerDied","Data":"a267558825ebcd679621af31c8ba543e4f61af277ae07cec3d55627cc7329eeb"}
Nov 02 18:50:40.549865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.549847    4956 scope.go:122] "RemoveContainer" containerID="71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab"
Nov 02 18:50:40.550299 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:40.550278    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab\": container with ID starting with 71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab not found: ID does not exist" containerID="71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab"
Nov 02 18:50:40.550392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.550307    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab"} err="failed to get container status \"71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab\": rpc error: code = NotFound desc = could not find container \"71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab\": container with ID starting with 71ab1bc0a9ffc663bbc5e406b43e0e5718845591e2c7e998a10c597dfe014eab not found: ID does not exist"
Nov 02 18:50:40.550392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.550324    4956 scope.go:122] "RemoveContainer" containerID="a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f"
Nov 02 18:50:40.559254 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.559231    4956 scope.go:122] "RemoveContainer" containerID="a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f"
Nov 02 18:50:40.559577 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:40.559552    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f\": container with ID starting with a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f not found: ID does not exist" containerID="a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f"
Nov 02 18:50:40.559652 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.559588    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f"} err="failed to get container status \"a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f\": rpc error: code = NotFound desc = could not find container \"a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f\": container with ID starting with a98d5669ec50c8ebe50535f646c7ad1703ec995480f5c7611bd74aa5d559a88f not found: ID does not exist"
Nov 02 18:50:40.620790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.620746    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvggj\" (UniqueName: \"kubernetes.io/projected/437e79f2-76bd-4221-a2f8-8bd890420692-kube-api-access-xvggj\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:40.701876 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.701846    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"]
Nov 02 18:50:40.703828 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.703803    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ceb1ae-74594bd85-w9hpc"]
Nov 02 18:50:40.810838 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.810801    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"]
Nov 02 18:50:40.812738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:40.812715    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ceb1ae-7848dc66b7-vlfcr"]
Nov 02 18:50:41.272890 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.272850    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"]
Nov 02 18:50:41.273163 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.273074    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26" podUID="ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" containerName="maas-api" containerID="cri-o://90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550" gracePeriod=30
Nov 02 18:50:41.520050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.520026    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 18:50:41.550765 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.550681    4956 generic.go:356] "Generic (PLEG): container finished" podID="ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" containerID="90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550" exitCode=0
Nov 02 18:50:41.550765 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.550722    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"
Nov 02 18:50:41.550964 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.550759    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26" event={"ID":"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4","Type":"ContainerDied","Data":"90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550"}
Nov 02 18:50:41.550964 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.550815    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26" event={"ID":"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4","Type":"ContainerDied","Data":"a2c2856e82dc948f1ff25329decf51bebdd909cecfe8a579c542e9cc50a4c76c"}
Nov 02 18:50:41.550964 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.550848    4956 scope.go:122] "RemoveContainer" containerID="90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550"
Nov 02 18:50:41.560052 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.560035    4956 scope.go:122] "RemoveContainer" containerID="90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550"
Nov 02 18:50:41.560375 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:41.560350    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550\": container with ID starting with 90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550 not found: ID does not exist" containerID="90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550"
Nov 02 18:50:41.560463 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.560384    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550"} err="failed to get container status \"90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550\": rpc error: code = NotFound desc = could not find container \"90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550\": container with ID starting with 90d7160eb00086e56b5df036f56c3e0e72057f82ba44d301d8aee79ca1df4550 not found: ID does not exist"
Nov 02 18:50:41.630443 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.630405    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-kube-api-access-ttq72\" (UniqueName: \"kubernetes.io/projected/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-kube-api-access-ttq72\") pod \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") "
Nov 02 18:50:41.630628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.630527    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls\") pod \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") "
Nov 02 18:50:41.630628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.630592    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-metrics-certs\") pod \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") "
Nov 02 18:50:41.630747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.630726    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-openshift-service-ca\") pod \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\" (UID: \"ed6e2db6-5a36-4c13-8fde-1ccb7db50de4\") "
Nov 02 18:50:41.631086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.631051    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-openshift-service-ca" pod "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" (UID: "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:41.632987 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.632959    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls" pod "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" (UID: "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:41.633340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.633318    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-kube-api-access-ttq72" pod "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" (UID: "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4"). InnerVolumeSpecName "kube-api-access-ttq72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:41.633461 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.633428    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-metrics-certs" pod "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" (UID: "ed6e2db6-5a36-4c13-8fde-1ccb7db50de4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:41.731712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.731673    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttq72\" (UniqueName: \"kubernetes.io/projected/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-kube-api-access-ttq72\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:41.731712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.731708    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:41.731942 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.731723    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:41.731942 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.731735    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:41.865744 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.865712    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f4460b-b73d-4949-827f-c59c14e5348d" path="/var/lib/kubelet/pods/21f4460b-b73d-4949-827f-c59c14e5348d/volumes"
Nov 02 18:50:41.866132 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.866104    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437e79f2-76bd-4221-a2f8-8bd890420692" path="/var/lib/kubelet/pods/437e79f2-76bd-4221-a2f8-8bd890420692/volumes"
Nov 02 18:50:41.969402 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.969369    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"]
Nov 02 18:50:41.974616 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:41.974581    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a099c3-5794c885cf-jjn26"]
Nov 02 18:50:43.864771 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:43.864738    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" path="/var/lib/kubelet/pods/ed6e2db6-5a36-4c13-8fde-1ccb7db50de4/volumes"
Nov 02 18:50:43.992366 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:43.992330    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"]
Nov 02 18:50:47.140790 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:47.140754    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"]
Nov 02 18:50:48.115302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:48.115258    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" podUID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:56.411669 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.411602    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 18:50:56.595291 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.595250    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-kube-api-access-zklqt\" (UniqueName: \"kubernetes.io/projected/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-kube-api-access-zklqt\") pod \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") "
Nov 02 18:50:56.595475 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.595310    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-plugins-config-volume\") pod \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\" (UID: \"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5\") "
Nov 02 18:50:56.595770 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.595745    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-plugins-config-volume" pod "2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" (UID: "2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:56.597660 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.597637    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-kube-api-access-zklqt" pod "2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" (UID: "2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5"). InnerVolumeSpecName "kube-api-access-zklqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:56.643063 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.643032    4956 generic.go:356] "Generic (PLEG): container finished" podID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" containerID="09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963" exitCode=0
Nov 02 18:50:56.643225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.643079    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"
Nov 02 18:50:56.643225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.643112    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" event={"ID":"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5","Type":"ContainerDied","Data":"09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963"}
Nov 02 18:50:56.643225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.643192    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss" event={"ID":"2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5","Type":"ContainerDied","Data":"18528e5f15bd0aaa57cf754ba76a21c090cb0787137cd0e1bbf53af809b315d1"}
Nov 02 18:50:56.643225 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.643214    4956 scope.go:122] "RemoveContainer" containerID="09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963"
Nov 02 18:50:56.653350 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.653322    4956 scope.go:122] "RemoveContainer" containerID="09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963"
Nov 02 18:50:56.653677 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:56.653656    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963\": container with ID starting with 09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963 not found: ID does not exist" containerID="09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963"
Nov 02 18:50:56.653752 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.653685    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963"} err="failed to get container status \"09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963\": rpc error: code = NotFound desc = could not find container \"09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963\": container with ID starting with 09929a020b8d585857c541e08549e85e200c781af8eb16d216276076be0d9963 not found: ID does not exist"
Nov 02 18:50:56.697426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.697357    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zklqt\" (UniqueName: \"kubernetes.io/projected/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-kube-api-access-zklqt\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:56.697426 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.697382    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:56.806213 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.806177    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"]
Nov 02 18:50:56.809249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:56.809226    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a099c3-5cf65b69c9-xb2ss"]
Nov 02 18:50:57.865767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:57.865733    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" path="/var/lib/kubelet/pods/2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5/volumes"
Nov 02 18:50:58.417046 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:58.417013    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"]
Nov 02 18:50:58.417322 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:58.417293    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy" containerID="cri-o://b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96" gracePeriod=30
Nov 02 18:50:59.013241 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.013203    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb" podUID="2d96b340-aed6-4f2e-9387-0f8fa9a27f33" containerName="main" containerID="cri-o://4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431" gracePeriod=45
Nov 02 18:50:59.058239 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:59.058194    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:50:59.271131 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.271058    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:50:59.322818 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.322780    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-home\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-home\") pod \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") "
Nov 02 18:50:59.322993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.322885    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tls-certs\") pod \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") "
Nov 02 18:50:59.322993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.322906    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-dshm\") pod \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") "
Nov 02 18:50:59.322993 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.322966    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-model-cache\") pod \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") "
Nov 02 18:50:59.323216 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.322995    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tmp-dir\") pod \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") "
Nov 02 18:50:59.323216 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.323016    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-kube-api-access-pcfvl\" (UniqueName: \"kubernetes.io/projected/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-kube-api-access-pcfvl\") pod \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\" (UID: \"2d96b340-aed6-4f2e-9387-0f8fa9a27f33\") "
Nov 02 18:50:59.323216 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.323045    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-home" pod "2d96b340-aed6-4f2e-9387-0f8fa9a27f33" (UID: "2d96b340-aed6-4f2e-9387-0f8fa9a27f33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:59.323366 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.323338    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-model-cache" pod "2d96b340-aed6-4f2e-9387-0f8fa9a27f33" (UID: "2d96b340-aed6-4f2e-9387-0f8fa9a27f33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:59.323432 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.323357    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tmp-dir" pod "2d96b340-aed6-4f2e-9387-0f8fa9a27f33" (UID: "2d96b340-aed6-4f2e-9387-0f8fa9a27f33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:59.323485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.323434    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:59.323485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.323454    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:59.323485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.323464    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:59.325434 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.325403    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tls-certs" pod "2d96b340-aed6-4f2e-9387-0f8fa9a27f33" (UID: "2d96b340-aed6-4f2e-9387-0f8fa9a27f33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:59.325434 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.325416    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-dshm" pod "2d96b340-aed6-4f2e-9387-0f8fa9a27f33" (UID: "2d96b340-aed6-4f2e-9387-0f8fa9a27f33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:59.325585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.325440    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-kube-api-access-pcfvl" pod "2d96b340-aed6-4f2e-9387-0f8fa9a27f33" (UID: "2d96b340-aed6-4f2e-9387-0f8fa9a27f33"). InnerVolumeSpecName "kube-api-access-pcfvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:59.424076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.424042    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:59.424076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.424069    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:59.424076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.424080    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcfvl\" (UniqueName: \"kubernetes.io/projected/2d96b340-aed6-4f2e-9387-0f8fa9a27f33-kube-api-access-pcfvl\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:59.662041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.661960    4956 generic.go:356] "Generic (PLEG): container finished" podID="2d96b340-aed6-4f2e-9387-0f8fa9a27f33" containerID="4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431" exitCode=0
Nov 02 18:50:59.662041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.662014    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb" event={"ID":"2d96b340-aed6-4f2e-9387-0f8fa9a27f33","Type":"ContainerDied","Data":"4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431"}
Nov 02 18:50:59.662041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.662041    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb" event={"ID":"2d96b340-aed6-4f2e-9387-0f8fa9a27f33","Type":"ContainerDied","Data":"58bc3b24ba45dc577974b76ddfb3aefb876d24812b2830edec14cee3ee030254"}
Nov 02 18:50:59.662328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.662058    4956 scope.go:122] "RemoveContainer" containerID="4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431"
Nov 02 18:50:59.662328 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.662013    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"
Nov 02 18:50:59.671585 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.671564    4956 scope.go:122] "RemoveContainer" containerID="4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431"
Nov 02 18:50:59.671822 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:50:59.671804    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431\": container with ID starting with 4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431 not found: ID does not exist" containerID="4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431"
Nov 02 18:50:59.671864 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.671831    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431"} err="failed to get container status \"4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431\": rpc error: code = NotFound desc = could not find container \"4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431\": container with ID starting with 4726ae21597ab4111f543b562a46963fb98f0e5be35ba22dc8b12a9ff8192431 not found: ID does not exist"
Nov 02 18:50:59.824893 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.824856    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"]
Nov 02 18:50:59.829658 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.829627    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/rate-test-model-f7aa2e-kserve-868647bfd9-d6hlb"]
Nov 02 18:50:59.865702 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:50:59.865668    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d96b340-aed6-4f2e-9387-0f8fa9a27f33" path="/var/lib/kubelet/pods/2d96b340-aed6-4f2e-9387-0f8fa9a27f33/volumes"
Nov 02 18:51:02.161473 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.161425    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml" podUID="1422e817-b4ec-47fd-849b-b368859b9dfa" containerName="main" containerID="cri-o://531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b" gracePeriod=45
Nov 02 18:51:02.411699 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.411636    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:51:02.449780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.449698    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-tmp-dir\") pod \"1422e817-b4ec-47fd-849b-b368859b9dfa\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") "
Nov 02 18:51:02.449780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.449759    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-model-cache\") pod \"1422e817-b4ec-47fd-849b-b368859b9dfa\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") "
Nov 02 18:51:02.450041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.449792    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1422e817-b4ec-47fd-849b-b368859b9dfa-kube-api-access-l2vp8\" (UniqueName: \"kubernetes.io/projected/1422e817-b4ec-47fd-849b-b368859b9dfa-kube-api-access-l2vp8\") pod \"1422e817-b4ec-47fd-849b-b368859b9dfa\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") "
Nov 02 18:51:02.450041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.449813    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1422e817-b4ec-47fd-849b-b368859b9dfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/1422e817-b4ec-47fd-849b-b368859b9dfa-tls-certs\") pod \"1422e817-b4ec-47fd-849b-b368859b9dfa\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") "
Nov 02 18:51:02.450041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.449872    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-home\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-home\") pod \"1422e817-b4ec-47fd-849b-b368859b9dfa\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") "
Nov 02 18:51:02.450041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.449886    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-dshm\") pod \"1422e817-b4ec-47fd-849b-b368859b9dfa\" (UID: \"1422e817-b4ec-47fd-849b-b368859b9dfa\") "
Nov 02 18:51:02.450292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.450071    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-model-cache" pod "1422e817-b4ec-47fd-849b-b368859b9dfa" (UID: "1422e817-b4ec-47fd-849b-b368859b9dfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:02.450292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.450109    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-tmp-dir" pod "1422e817-b4ec-47fd-849b-b368859b9dfa" (UID: "1422e817-b4ec-47fd-849b-b368859b9dfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:02.450292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.450200    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-home" pod "1422e817-b4ec-47fd-849b-b368859b9dfa" (UID: "1422e817-b4ec-47fd-849b-b368859b9dfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:02.450404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.450380    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:02.450404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.450398    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:02.450477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.450407    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:02.452301 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.452279    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-dshm" pod "1422e817-b4ec-47fd-849b-b368859b9dfa" (UID: "1422e817-b4ec-47fd-849b-b368859b9dfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:02.452385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.452299    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1422e817-b4ec-47fd-849b-b368859b9dfa-kube-api-access-l2vp8" pod "1422e817-b4ec-47fd-849b-b368859b9dfa" (UID: "1422e817-b4ec-47fd-849b-b368859b9dfa"). InnerVolumeSpecName "kube-api-access-l2vp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:02.452385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.452361    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1422e817-b4ec-47fd-849b-b368859b9dfa-tls-certs" pod "1422e817-b4ec-47fd-849b-b368859b9dfa" (UID: "1422e817-b4ec-47fd-849b-b368859b9dfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:51:02.551056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.551013    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1422e817-b4ec-47fd-849b-b368859b9dfa-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:02.551056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.551056    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2vp8\" (UniqueName: \"kubernetes.io/projected/1422e817-b4ec-47fd-849b-b368859b9dfa-kube-api-access-l2vp8\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:02.551294 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.551072    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1422e817-b4ec-47fd-849b-b368859b9dfa-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:02.680389 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.680296    4956 generic.go:356] "Generic (PLEG): container finished" podID="1422e817-b4ec-47fd-849b-b368859b9dfa" containerID="531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b" exitCode=0
Nov 02 18:51:02.680389 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.680350    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"
Nov 02 18:51:02.680628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.680389    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml" event={"ID":"1422e817-b4ec-47fd-849b-b368859b9dfa","Type":"ContainerDied","Data":"531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b"}
Nov 02 18:51:02.680628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.680435    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml" event={"ID":"1422e817-b4ec-47fd-849b-b368859b9dfa","Type":"ContainerDied","Data":"f89ceed804307b1b3bdd677c22afdd7391c60c91b21e0af24c8e69f4ced815fc"}
Nov 02 18:51:02.680628 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.680454    4956 scope.go:122] "RemoveContainer" containerID="531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b"
Nov 02 18:51:02.690586 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.690567    4956 scope.go:122] "RemoveContainer" containerID="531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b"
Nov 02 18:51:02.690881 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:51:02.690862    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b\": container with ID starting with 531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b not found: ID does not exist" containerID="531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b"
Nov 02 18:51:02.690948 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.690895    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b"} err="failed to get container status \"531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b\": rpc error: code = NotFound desc = could not find container \"531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b\": container with ID starting with 531f12b9617e5286aed52c685a05964898ca61cb5c7fe2ee4a269224353f6f1b not found: ID does not exist"
Nov 02 18:51:02.842146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.842083    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"]
Nov 02 18:51:02.844181 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:02.844160    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2a1bcc/rate-test-model-2a1bcc-kserve-5dd9cf9fc-qx4ml"]
Nov 02 18:51:03.015249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.015177    4956 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:51:03.015249 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.015240    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:51:03.672629 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.672603    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 18:51:03.688059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.687986    4956 generic.go:356] "Generic (PLEG): container finished" podID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerID="b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96" exitCode=0
Nov 02 18:51:03.688059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.688047    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"
Nov 02 18:51:03.688059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.688047    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" event={"ID":"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3","Type":"ContainerDied","Data":"b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96"}
Nov 02 18:51:03.688297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.688157    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w" event={"ID":"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3","Type":"ContainerDied","Data":"de3650e130e3384368251ac5218699ad53ebb0061b3c0785f7c0298391581f95"}
Nov 02 18:51:03.688297 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.688189    4956 scope.go:122] "RemoveContainer" containerID="b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96"
Nov 02 18:51:03.697093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.697073    4956 scope.go:122] "RemoveContainer" containerID="bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0"
Nov 02 18:51:03.705281 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.705261    4956 scope.go:122] "RemoveContainer" containerID="b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96"
Nov 02 18:51:03.705539 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:51:03.705517    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96\": container with ID starting with b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96 not found: ID does not exist" containerID="b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96"
Nov 02 18:51:03.705595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.705547    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96"} err="failed to get container status \"b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96\": rpc error: code = NotFound desc = could not find container \"b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96\": container with ID starting with b95518b56b18f7747bb085709bb1b370bc3aebeb4a89d0f5ddcbd9b58f0eaa96 not found: ID does not exist"
Nov 02 18:51:03.705595 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.705566    4956 scope.go:122] "RemoveContainer" containerID="bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0"
Nov 02 18:51:03.705765 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:51:03.705751    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0\": container with ID starting with bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0 not found: ID does not exist" containerID="bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0"
Nov 02 18:51:03.705822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.705767    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0"} err="failed to get container status \"bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0\": rpc error: code = NotFound desc = could not find container \"bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0\": container with ID starting with bc484b65190809afbbc7305716e86cd80f74312d0eeeced55d8d9626e62e3ed0 not found: ID does not exist"
Nov 02 18:51:03.760652 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760617    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-certs\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.760831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760671    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-socket\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.760831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760712    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-podinfo\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.760831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760735    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-kube-api-access-5tct5\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-kube-api-access-5tct5\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.760831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760786    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-token\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-token\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.760831 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760827    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-credential-socket\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.761108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760860    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-data\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.761108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760906    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istiod-ca-cert\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.761108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760943    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-envoy\") pod \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\" (UID: \"ee9ba38f-17de-42f7-9ee2-addf0fa5efb3\") "
Nov 02 18:51:03.761108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.760952    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-certs" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:03.761108 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761059    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-socket" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:03.761421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761205    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-data" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:03.761421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761219    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-credential-socket" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:03.761421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761358    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.761421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761380    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.761421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761394    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.761421 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761405    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.761905 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.761417    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istiod-ca-cert" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:51:03.763193 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.763166    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-podinfo" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:51:03.763295 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.763171    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-token" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:03.763709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.763689    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-kube-api-access-5tct5" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "kube-api-access-5tct5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:03.763709 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.763697    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-envoy" pod "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" (UID: "ee9ba38f-17de-42f7-9ee2-addf0fa5efb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:03.861867 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.861826    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.861867 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.861861    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tct5\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-kube-api-access-5tct5\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.862069 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.861876    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.862069 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.861888    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.862069 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.861903    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:03.864868 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:03.864838    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1422e817-b4ec-47fd-849b-b368859b9dfa" path="/var/lib/kubelet/pods/1422e817-b4ec-47fd-849b-b368859b9dfa/volumes"
Nov 02 18:51:04.101334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:04.101296    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"]
Nov 02 18:51:04.105269 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:04.105248    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-mlg7w"]
Nov 02 18:51:05.864331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:05.864297    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" path="/var/lib/kubelet/pods/ee9ba38f-17de-42f7-9ee2-addf0fa5efb3/volumes"
Nov 02 18:51:28.992597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:28.992561    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"]
Nov 02 18:51:29.052225 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:51:29.052187    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:51:33.993559 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:33.993515    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" podUID="49ea85e4-0611-4640-8172-95584a79424e" containerName="payload-processing" containerID="cri-o://53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788" gracePeriod=25
Nov 02 18:51:35.648056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:35.648016    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" podUID="49ea85e4-0611-4640-8172-95584a79424e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:51:45.648086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:45.648046    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" podUID="49ea85e4-0611-4640-8172-95584a79424e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:51:54.244821 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.244793    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:51:54.339404 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.339312    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ea85e4-0611-4640-8172-95584a79424e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea85e4-0611-4640-8172-95584a79424e-plugins-config-volume\") pod \"49ea85e4-0611-4640-8172-95584a79424e\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") "
Nov 02 18:51:54.339598 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.339417    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ea85e4-0611-4640-8172-95584a79424e-kube-api-access-vr4lf\" (UniqueName: \"kubernetes.io/projected/49ea85e4-0611-4640-8172-95584a79424e-kube-api-access-vr4lf\") pod \"49ea85e4-0611-4640-8172-95584a79424e\" (UID: \"49ea85e4-0611-4640-8172-95584a79424e\") "
Nov 02 18:51:54.339762 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.339736    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ea85e4-0611-4640-8172-95584a79424e-plugins-config-volume" pod "49ea85e4-0611-4640-8172-95584a79424e" (UID: "49ea85e4-0611-4640-8172-95584a79424e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:51:54.341614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.341582    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea85e4-0611-4640-8172-95584a79424e-kube-api-access-vr4lf" pod "49ea85e4-0611-4640-8172-95584a79424e" (UID: "49ea85e4-0611-4640-8172-95584a79424e"). InnerVolumeSpecName "kube-api-access-vr4lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:54.441169 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.441098    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea85e4-0611-4640-8172-95584a79424e-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:54.441169 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.441163    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr4lf\" (UniqueName: \"kubernetes.io/projected/49ea85e4-0611-4640-8172-95584a79424e-kube-api-access-vr4lf\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:54.952809 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.952775    4956 generic.go:356] "Generic (PLEG): container finished" podID="49ea85e4-0611-4640-8172-95584a79424e" containerID="53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788" exitCode=0
Nov 02 18:51:54.953029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.952819    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"
Nov 02 18:51:54.953029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.952872    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" event={"ID":"49ea85e4-0611-4640-8172-95584a79424e","Type":"ContainerDied","Data":"53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788"}
Nov 02 18:51:54.953029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.952924    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5" event={"ID":"49ea85e4-0611-4640-8172-95584a79424e","Type":"ContainerDied","Data":"6da0a96cbf5d0bcf7fd70e818152ac0a59c6925040f5516a6ffda4380d92a927"}
Nov 02 18:51:54.953029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.952950    4956 scope.go:122] "RemoveContainer" containerID="53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788"
Nov 02 18:51:54.962685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.962668    4956 scope.go:122] "RemoveContainer" containerID="53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788"
Nov 02 18:51:54.962995 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:51:54.962960    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788\": container with ID starting with 53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788 not found: ID does not exist" containerID="53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788"
Nov 02 18:51:54.963096 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:54.963001    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788"} err="failed to get container status \"53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788\": rpc error: code = NotFound desc = could not find container \"53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788\": container with ID starting with 53dc468a00ccc0c63b6e278140d159d397962e0e8bc6d986ddaa1a7afb608788 not found: ID does not exist"
Nov 02 18:51:55.115151 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:55.115103    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"]
Nov 02 18:51:55.164720 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:55.164684    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f7aa2e-6b6ddc9fff-9lcg5"]
Nov 02 18:51:55.862976 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:55.862942    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ea85e4-0611-4640-8172-95584a79424e" path="/var/lib/kubelet/pods/49ea85e4-0611-4640-8172-95584a79424e/volumes"
Nov 02 18:51:58.962458 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:58.962421    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"]
Nov 02 18:51:59.051855 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:51:59.051817    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:51:59.076046 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:51:59.076010    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"]
Nov 02 18:52:03.963614 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:03.963514    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" podUID="e80a322f-cc91-4e81-9607-0698741dec9f" containerName="payload-processing" containerID="cri-o://0a3a21bb32196f363b11f606a68b985030a984076b9eef0681a5d6b8dfe3d828" gracePeriod=25
Nov 02 18:52:04.076635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:04.076584    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" podUID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" containerName="payload-pre-processing" containerID="cri-o://23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841" gracePeriod=25
Nov 02 18:52:10.686282 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:10.686246    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"]
Nov 02 18:52:10.686848 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:10.686457    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9" podUID="6c02d40d-c7bb-4b17-8c06-5570d14496a4" containerName="maas-api" containerID="cri-o://fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8" gracePeriod=30
Nov 02 18:52:10.938392 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:10.938326    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:52:11.005144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.005105    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls\") pod \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") "
Nov 02 18:52:11.005144 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.005155    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c02d40d-c7bb-4b17-8c06-5570d14496a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02d40d-c7bb-4b17-8c06-5570d14496a4-openshift-service-ca\") pod \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") "
Nov 02 18:52:11.005418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.005221    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-metrics-certs\") pod \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") "
Nov 02 18:52:11.005418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.005328    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c02d40d-c7bb-4b17-8c06-5570d14496a4-kube-api-access-vbcjv\" (UniqueName: \"kubernetes.io/projected/6c02d40d-c7bb-4b17-8c06-5570d14496a4-kube-api-access-vbcjv\") pod \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\" (UID: \"6c02d40d-c7bb-4b17-8c06-5570d14496a4\") "
Nov 02 18:52:11.005603 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.005579    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c02d40d-c7bb-4b17-8c06-5570d14496a4-openshift-service-ca" pod "6c02d40d-c7bb-4b17-8c06-5570d14496a4" (UID: "6c02d40d-c7bb-4b17-8c06-5570d14496a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:11.005713 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.005695    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c02d40d-c7bb-4b17-8c06-5570d14496a4-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:11.007475 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.007443    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls" pod "6c02d40d-c7bb-4b17-8c06-5570d14496a4" (UID: "6c02d40d-c7bb-4b17-8c06-5570d14496a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:11.007594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.007524    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-metrics-certs" pod "6c02d40d-c7bb-4b17-8c06-5570d14496a4" (UID: "6c02d40d-c7bb-4b17-8c06-5570d14496a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:11.007594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.007567    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c02d40d-c7bb-4b17-8c06-5570d14496a4-kube-api-access-vbcjv" pod "6c02d40d-c7bb-4b17-8c06-5570d14496a4" (UID: "6c02d40d-c7bb-4b17-8c06-5570d14496a4"). InnerVolumeSpecName "kube-api-access-vbcjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:11.032032 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.031999    4956 generic.go:356] "Generic (PLEG): container finished" podID="6c02d40d-c7bb-4b17-8c06-5570d14496a4" containerID="fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8" exitCode=0
Nov 02 18:52:11.032203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.032033    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9" event={"ID":"6c02d40d-c7bb-4b17-8c06-5570d14496a4","Type":"ContainerDied","Data":"fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8"}
Nov 02 18:52:11.032203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.032050    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"
Nov 02 18:52:11.032203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.032090    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9" event={"ID":"6c02d40d-c7bb-4b17-8c06-5570d14496a4","Type":"ContainerDied","Data":"9d084057d34bd2729a18cfd49730aabe00142d9ede2369b5a300641db93dcdac"}
Nov 02 18:52:11.032203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.032113    4956 scope.go:122] "RemoveContainer" containerID="fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8"
Nov 02 18:52:11.041358 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.041341    4956 scope.go:122] "RemoveContainer" containerID="fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8"
Nov 02 18:52:11.041631 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:52:11.041610    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8\": container with ID starting with fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8 not found: ID does not exist" containerID="fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8"
Nov 02 18:52:11.041695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.041638    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8"} err="failed to get container status \"fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8\": rpc error: code = NotFound desc = could not find container \"fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8\": container with ID starting with fffcdbb564db2a6cd94c95f4601f53426d2348318b5ace9c9f9c4e2b274ac7b8 not found: ID does not exist"
Nov 02 18:52:11.106525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.106495    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbcjv\" (UniqueName: \"kubernetes.io/projected/6c02d40d-c7bb-4b17-8c06-5570d14496a4-kube-api-access-vbcjv\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:11.106525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.106523    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:11.106747 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.106537    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c02d40d-c7bb-4b17-8c06-5570d14496a4-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:11.161970 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.161929    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"]
Nov 02 18:52:11.166786 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.166749    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a1bcc-fd789bf5-ph2h9"]
Nov 02 18:52:11.863539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:11.863507    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c02d40d-c7bb-4b17-8c06-5570d14496a4" path="/var/lib/kubelet/pods/6c02d40d-c7bb-4b17-8c06-5570d14496a4/volumes"
Nov 02 18:52:13.484955 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:13.484912    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" podUID="e80a322f-cc91-4e81-9607-0698741dec9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:13.484955 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:13.484937    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" podUID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:23.485155 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:23.485096    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" podUID="e80a322f-cc91-4e81-9607-0698741dec9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:23.485576 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:23.485097    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" podUID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:24.098670 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.098640    4956 generic.go:356] "Generic (PLEG): container finished" podID="e80a322f-cc91-4e81-9607-0698741dec9f" containerID="0a3a21bb32196f363b11f606a68b985030a984076b9eef0681a5d6b8dfe3d828" exitCode=0
Nov 02 18:52:24.098807 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.098708    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" event={"ID":"e80a322f-cc91-4e81-9607-0698741dec9f","Type":"ContainerDied","Data":"0a3a21bb32196f363b11f606a68b985030a984076b9eef0681a5d6b8dfe3d828"}
Nov 02 18:52:24.222043 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.221978    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:52:24.338727 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.338701    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:52:24.339095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.339075    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80a322f-cc91-4e81-9607-0698741dec9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80a322f-cc91-4e81-9607-0698741dec9f-plugins-config-volume\") pod \"e80a322f-cc91-4e81-9607-0698741dec9f\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") "
Nov 02 18:52:24.339231 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.339218    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80a322f-cc91-4e81-9607-0698741dec9f-kube-api-access-jfzkh\" (UniqueName: \"kubernetes.io/projected/e80a322f-cc91-4e81-9607-0698741dec9f-kube-api-access-jfzkh\") pod \"e80a322f-cc91-4e81-9607-0698741dec9f\" (UID: \"e80a322f-cc91-4e81-9607-0698741dec9f\") "
Nov 02 18:52:24.339473 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.339452    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80a322f-cc91-4e81-9607-0698741dec9f-plugins-config-volume" pod "e80a322f-cc91-4e81-9607-0698741dec9f" (UID: "e80a322f-cc91-4e81-9607-0698741dec9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:24.341650 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.341623    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80a322f-cc91-4e81-9607-0698741dec9f-kube-api-access-jfzkh" pod "e80a322f-cc91-4e81-9607-0698741dec9f" (UID: "e80a322f-cc91-4e81-9607-0698741dec9f"). InnerVolumeSpecName "kube-api-access-jfzkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:24.440380 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.440344    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-kube-api-access-jgs7g\" (UniqueName: \"kubernetes.io/projected/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-kube-api-access-jgs7g\") pod \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") "
Nov 02 18:52:24.440567 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.440432    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-plugins-config-volume\") pod \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\" (UID: \"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2\") "
Nov 02 18:52:24.440767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.440746    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80a322f-cc91-4e81-9607-0698741dec9f-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:24.440806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.440775    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfzkh\" (UniqueName: \"kubernetes.io/projected/e80a322f-cc91-4e81-9607-0698741dec9f-kube-api-access-jfzkh\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:24.440806 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.440780    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-plugins-config-volume" pod "8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" (UID: "8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:24.442707 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.442688    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-kube-api-access-jgs7g" pod "8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" (UID: "8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2"). InnerVolumeSpecName "kube-api-access-jgs7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:24.542192 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.542084    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgs7g\" (UniqueName: \"kubernetes.io/projected/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-kube-api-access-jgs7g\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:24.542192 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:24.542143    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.104887 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.104845    4956 generic.go:356] "Generic (PLEG): container finished" podID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" containerID="23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841" exitCode=0
Nov 02 18:52:25.105095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.104894    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"
Nov 02 18:52:25.105095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.104936    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" event={"ID":"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2","Type":"ContainerDied","Data":"23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841"}
Nov 02 18:52:25.105095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.104992    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx" event={"ID":"8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2","Type":"ContainerDied","Data":"45472f0e8a54fee86d2362448599741ce811de3eeb5832bb882c3e6a97fa4459"}
Nov 02 18:52:25.105095 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.105019    4956 scope.go:122] "RemoveContainer" containerID="23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841"
Nov 02 18:52:25.106503 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.106285    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq" event={"ID":"e80a322f-cc91-4e81-9607-0698741dec9f","Type":"ContainerDied","Data":"800caf437c0206bd9f65888334ecb1dd111a09eed5c6dcf678231894789364ee"}
Nov 02 18:52:25.106503 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.106343    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"
Nov 02 18:52:25.121090 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.121064    4956 scope.go:122] "RemoveContainer" containerID="23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841"
Nov 02 18:52:25.121434 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:52:25.121401    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841\": container with ID starting with 23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841 not found: ID does not exist" containerID="23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841"
Nov 02 18:52:25.121532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.121464    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841"} err="failed to get container status \"23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841\": rpc error: code = NotFound desc = could not find container \"23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841\": container with ID starting with 23212ec40f2a3d006133b0e570cf36507b9e95a61c9530529fd22c2ee2784841 not found: ID does not exist"
Nov 02 18:52:25.121532 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.121493    4956 scope.go:122] "RemoveContainer" containerID="0a3a21bb32196f363b11f606a68b985030a984076b9eef0681a5d6b8dfe3d828"
Nov 02 18:52:25.266667 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.266589    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"]
Nov 02 18:52:25.316084 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.316049    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2a1bcc-5f8f8dbf9f-xhbnx"]
Nov 02 18:52:25.377353 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.377320    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"]
Nov 02 18:52:25.380047 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.380023    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a1bcc-555cc7dbb5-5wnqq"]
Nov 02 18:52:25.863678 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.863647    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" path="/var/lib/kubelet/pods/8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2/volumes"
Nov 02 18:52:25.864079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:25.863993    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80a322f-cc91-4e81-9607-0698741dec9f" path="/var/lib/kubelet/pods/e80a322f-cc91-4e81-9607-0698741dec9f/volumes"
Nov 02 18:52:29.054606 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:52:29.054570    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:52:34.121269 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:34.121204    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"]
Nov 02 18:52:34.227904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:34.227866    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"]
Nov 02 18:52:39.122495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:39.122445    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" podUID="869c0472-aebf-4527-a67b-703dd58994c0" containerName="payload-processing" containerID="cri-o://ad8db7a6ec475ae95e048c3744b70dadb1e894c3f8d0f76d50164a1c323daede" gracePeriod=25
Nov 02 18:52:39.228242 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:39.228199    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" podUID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" containerName="payload-pre-processing" containerID="cri-o://ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf" gracePeriod=25
Nov 02 18:52:48.185572 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:48.185533    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" podUID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:48.186055 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:48.185539    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" podUID="869c0472-aebf-4527-a67b-703dd58994c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:50.651066 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:50.651040    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:52:50.659573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:50.659548    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:52:58.185761 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:58.185720    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" podUID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:58.185761 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:58.185744    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" podUID="869c0472-aebf-4527-a67b-703dd58994c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:52:59.052253 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:52:59.052218    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:52:59.274692 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.274463    4956 generic.go:356] "Generic (PLEG): container finished" podID="869c0472-aebf-4527-a67b-703dd58994c0" containerID="ad8db7a6ec475ae95e048c3744b70dadb1e894c3f8d0f76d50164a1c323daede" exitCode=0
Nov 02 18:52:59.274692 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.274562    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" event={"ID":"869c0472-aebf-4527-a67b-703dd58994c0","Type":"ContainerDied","Data":"ad8db7a6ec475ae95e048c3744b70dadb1e894c3f8d0f76d50164a1c323daede"}
Nov 02 18:52:59.389892 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.389861    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 18:52:59.467923 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.467889    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869c0472-aebf-4527-a67b-703dd58994c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869c0472-aebf-4527-a67b-703dd58994c0-plugins-config-volume\") pod \"869c0472-aebf-4527-a67b-703dd58994c0\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") "
Nov 02 18:52:59.468113 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.468033    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869c0472-aebf-4527-a67b-703dd58994c0-kube-api-access-p2cnq\" (UniqueName: \"kubernetes.io/projected/869c0472-aebf-4527-a67b-703dd58994c0-kube-api-access-p2cnq\") pod \"869c0472-aebf-4527-a67b-703dd58994c0\" (UID: \"869c0472-aebf-4527-a67b-703dd58994c0\") "
Nov 02 18:52:59.468362 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.468337    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869c0472-aebf-4527-a67b-703dd58994c0-plugins-config-volume" pod "869c0472-aebf-4527-a67b-703dd58994c0" (UID: "869c0472-aebf-4527-a67b-703dd58994c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:59.470272 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.470248    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869c0472-aebf-4527-a67b-703dd58994c0-kube-api-access-p2cnq" pod "869c0472-aebf-4527-a67b-703dd58994c0" (UID: "869c0472-aebf-4527-a67b-703dd58994c0"). InnerVolumeSpecName "kube-api-access-p2cnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:59.484891 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.484861    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 18:52:59.568978 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.568937    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-kube-api-access-pfgrm\" (UniqueName: \"kubernetes.io/projected/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-kube-api-access-pfgrm\") pod \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") "
Nov 02 18:52:59.569207 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.569032    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-plugins-config-volume\") pod \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\" (UID: \"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8\") "
Nov 02 18:52:59.569468 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.569443    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-plugins-config-volume" pod "0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" (UID: "0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:59.569515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.569462    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2cnq\" (UniqueName: \"kubernetes.io/projected/869c0472-aebf-4527-a67b-703dd58994c0-kube-api-access-p2cnq\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:59.569515 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.569483    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869c0472-aebf-4527-a67b-703dd58994c0-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:59.571312 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.571293    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-kube-api-access-pfgrm" pod "0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" (UID: "0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8"). InnerVolumeSpecName "kube-api-access-pfgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:59.670767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.670656    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgrm\" (UniqueName: \"kubernetes.io/projected/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-kube-api-access-pfgrm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:59.670767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:52:59.670710    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:00.280365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.280332    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"
Nov 02 18:53:00.280858 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.280359    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl" event={"ID":"869c0472-aebf-4527-a67b-703dd58994c0","Type":"ContainerDied","Data":"fb3edf119bf9a60668b8f77a2001d6eb9a63ba622d3256a628da7b153d87a592"}
Nov 02 18:53:00.280858 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.280423    4956 scope.go:122] "RemoveContainer" containerID="ad8db7a6ec475ae95e048c3744b70dadb1e894c3f8d0f76d50164a1c323daede"
Nov 02 18:53:00.282302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.282275    4956 generic.go:356] "Generic (PLEG): container finished" podID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" containerID="ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf" exitCode=0
Nov 02 18:53:00.282477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.282333    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" event={"ID":"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8","Type":"ContainerDied","Data":"ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf"}
Nov 02 18:53:00.282477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.282361    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"
Nov 02 18:53:00.282477 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.282381    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb" event={"ID":"0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8","Type":"ContainerDied","Data":"a2ec98de314de5f1c40cd722775e39e0a18dfd1d4c02bc214fef645fafb1adc9"}
Nov 02 18:53:00.289881 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.289867    4956 scope.go:122] "RemoveContainer" containerID="ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf"
Nov 02 18:53:00.297870 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.297849    4956 scope.go:122] "RemoveContainer" containerID="ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf"
Nov 02 18:53:00.298140 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:53:00.298106    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf\": container with ID starting with ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf not found: ID does not exist" containerID="ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf"
Nov 02 18:53:00.298189 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.298156    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf"} err="failed to get container status \"ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf\": rpc error: code = NotFound desc = could not find container \"ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf\": container with ID starting with ab10e42a6b8a61da59b14944c3c83fc8100a23442c76b94b8befb87502c723cf not found: ID does not exist"
Nov 02 18:53:00.443900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.443816    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"]
Nov 02 18:53:00.447575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.447546    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6e3175-6746bbcc67-2s7wb"]
Nov 02 18:53:00.555902 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.555867    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"]
Nov 02 18:53:00.559557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:00.559531    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6e3175-76b77996c-l89pl"]
Nov 02 18:53:01.863539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:01.863504    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" path="/var/lib/kubelet/pods/0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8/volumes"
Nov 02 18:53:01.863930 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:01.863875    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869c0472-aebf-4527-a67b-703dd58994c0" path="/var/lib/kubelet/pods/869c0472-aebf-4527-a67b-703dd58994c0/volumes"
Nov 02 18:53:05.654076 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:05.654038    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"]
Nov 02 18:53:05.654573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:05.654277    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy" containerID="cri-o://7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a" gracePeriod=30
Nov 02 18:53:10.225733 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:10.225696    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5c9b9f7776-2hch9_89a58678-30c1-4403-81bd-9818ec081d07/manager/0.log"
Nov 02 18:53:10.705613 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:10.705579    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fvpqg_e85ee5b1-3fcf-4372-b108-0c7ec63511fa/manager/0.log"
Nov 02 18:53:10.902008 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:10.901983    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 18:53:11.084038 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.083991    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e764791c-08d9-4913-8496-9f0f0c566e98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764791c-08d9-4913-8496-9f0f0c566e98-istiod-ca-cert\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084098    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-envoy\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084193    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-kube-api-access-8l9dh\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-kube-api-access-8l9dh\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084232    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-certs\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084261    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-istio-token\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-istio-token\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084324 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084305    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-data\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084341    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-credential-socket\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084411    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e764791c-08d9-4913-8496-9f0f0c566e98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764791c-08d9-4913-8496-9f0f0c566e98-istio-podinfo\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.084710 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.084465    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-socket\") pod \"e764791c-08d9-4913-8496-9f0f0c566e98\" (UID: \"e764791c-08d9-4913-8496-9f0f0c566e98\") "
Nov 02 18:53:11.085203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.085175    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-socket" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:53:11.085506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.085483    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-certs" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:53:11.088511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.087330    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e764791c-08d9-4913-8496-9f0f0c566e98-istiod-ca-cert" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:53:11.088511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.088374    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-credential-socket" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:53:11.088511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.088424    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-data" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:53:11.091591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.091550    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-istio-token" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:53:11.091712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.091585    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-envoy" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:53:11.091712 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.091592    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-kube-api-access-8l9dh" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "kube-api-access-8l9dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:53:11.091942 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.091922    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e764791c-08d9-4913-8496-9f0f0c566e98-istio-podinfo" pod "e764791c-08d9-4913-8496-9f0f0c566e98" (UID: "e764791c-08d9-4913-8496-9f0f0c566e98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:53:11.185602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185569    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e764791c-08d9-4913-8496-9f0f0c566e98-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185596    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185609    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9dh\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-kube-api-access-8l9dh\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185619    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185628    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e764791c-08d9-4913-8496-9f0f0c566e98-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185637    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185646    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185654    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e764791c-08d9-4913-8496-9f0f0c566e98-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.185862 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.185662    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e764791c-08d9-4913-8496-9f0f0c566e98-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:53:11.341113 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.341017    4956 generic.go:356] "Generic (PLEG): container finished" podID="e764791c-08d9-4913-8496-9f0f0c566e98" containerID="7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a" exitCode=0
Nov 02 18:53:11.341594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.341104    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" event={"ID":"e764791c-08d9-4913-8496-9f0f0c566e98","Type":"ContainerDied","Data":"7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a"}
Nov 02 18:53:11.341594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.341070    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"
Nov 02 18:53:11.341594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.341192    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv" event={"ID":"e764791c-08d9-4913-8496-9f0f0c566e98","Type":"ContainerDied","Data":"e6689b688a8a58d0985cb28175dabcb87651e9c7976134bee9f54d5e06a142d2"}
Nov 02 18:53:11.341594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.341217    4956 scope.go:122] "RemoveContainer" containerID="7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a"
Nov 02 18:53:11.355672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.355646    4956 scope.go:122] "RemoveContainer" containerID="7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a"
Nov 02 18:53:11.356028 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:53:11.355977    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a\": container with ID starting with 7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a not found: ID does not exist" containerID="7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a"
Nov 02 18:53:11.356678 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.356474    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a"} err="failed to get container status \"7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a\": rpc error: code = NotFound desc = could not find container \"7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a\": container with ID starting with 7ae9bc13b617cd9bd057a6f9344951d582065ffa2e901c2033a43e2b38e6041a not found: ID does not exist"
Nov 02 18:53:11.503092 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.503046    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"]
Nov 02 18:53:11.506365 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.506339    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-wbbdv"]
Nov 02 18:53:11.864409 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:11.864377    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" path="/var/lib/kubelet/pods/e764791c-08d9-4913-8496-9f0f0c566e98/volumes"
Nov 02 18:53:24.603849 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.603814    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"]
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604246    4956 cpu_manager.go:414] "containerMap: removing container" podUID="869c0472-aebf-4527-a67b-703dd58994c0" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604260    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869c0472-aebf-4527-a67b-703dd58994c0" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604281    4956 cpu_manager.go:414] "containerMap: removing container" podUID="21f4460b-b73d-4949-827f-c59c14e5348d" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604286    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f4460b-b73d-4949-827f-c59c14e5348d" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604295    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604302    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604313    4956 cpu_manager.go:414] "containerMap: removing container" podUID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" containerName="payload-pre-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604318    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" containerName="payload-pre-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604325    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" containerName="payload-pre-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604330    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" containerName="payload-pre-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604341    4956 cpu_manager.go:414] "containerMap: removing container" podUID="6c02d40d-c7bb-4b17-8c06-5570d14496a4" containerName="maas-api"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604345    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c02d40d-c7bb-4b17-8c06-5570d14496a4" containerName="maas-api"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604351    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604357    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604365    4956 cpu_manager.go:414] "containerMap: removing container" podUID="437e79f2-76bd-4221-a2f8-8bd890420692" containerName="payload-pre-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604369    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437e79f2-76bd-4221-a2f8-8bd890420692" containerName="payload-pre-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604382    4956 cpu_manager.go:414] "containerMap: removing container" podUID="49ea85e4-0611-4640-8172-95584a79424e" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604389    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ea85e4-0611-4640-8172-95584a79424e" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604399    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1422e817-b4ec-47fd-849b-b368859b9dfa" containerName="main"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604404    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1422e817-b4ec-47fd-849b-b368859b9dfa" containerName="main"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604410    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e80a322f-cc91-4e81-9607-0698741dec9f" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604415    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80a322f-cc91-4e81-9607-0698741dec9f" containerName="payload-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604420    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604424    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604432    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604436    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604441    4956 cpu_manager.go:414] "containerMap: removing container" podUID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" containerName="payload-pre-processing"
Nov 02 18:53:24.604433 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604446    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" containerName="payload-pre-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604452    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" containerName="maas-api"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604459    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" containerName="maas-api"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604474    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8ce42215-7e35-4588-9083-54c2c9611934" containerName="maas-api"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604479    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce42215-7e35-4588-9083-54c2c9611934" containerName="maas-api"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604489    4956 cpu_manager.go:414] "containerMap: removing container" podUID="2d96b340-aed6-4f2e-9387-0f8fa9a27f33" containerName="main"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604494    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d96b340-aed6-4f2e-9387-0f8fa9a27f33" containerName="main"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604499    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604504    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604511    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604515    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604578    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c02d40d-c7bb-4b17-8c06-5570d14496a4" containerName="maas-api"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604590    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604595    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="437e79f2-76bd-4221-a2f8-8bd890420692" containerName="payload-pre-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604601    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ea85e4-0611-4640-8172-95584a79424e" containerName="payload-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604606    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1422e817-b4ec-47fd-849b-b368859b9dfa" containerName="main"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604612    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80a322f-cc91-4e81-9607-0698741dec9f" containerName="payload-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604618    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e764791c-08d9-4913-8496-9f0f0c566e98" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604627    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6389d6f-249f-4998-8760-30c38316ba91" containerName="main"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604637    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="0601d8b3-f4a6-4680-b5d6-a0b3b1d7daa8" containerName="payload-pre-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604642    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6e2db6-5a36-4c13-8fde-1ccb7db50de4" containerName="maas-api"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604647    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce42215-7e35-4588-9083-54c2c9611934" containerName="maas-api"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604653    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d96b340-aed6-4f2e-9387-0f8fa9a27f33" containerName="main"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604657    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e92823d-fd4e-40fa-a5a4-c4aab1ccdcca" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604663    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9ba38f-17de-42f7-9ee2-addf0fa5efb3" containerName="istio-proxy"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604668    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="869c0472-aebf-4527-a67b-703dd58994c0" containerName="payload-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604675    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f4460b-b73d-4949-827f-c59c14e5348d" containerName="payload-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604679    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6de910d-3a7e-47f8-957f-3d2735c9bf27" containerName="main"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604684    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb26302-c3fd-42a0-8ea6-2ce10dc1fcd5" containerName="payload-pre-processing"
Nov 02 18:53:24.605343 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.604689    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8fb3e7-5cfc-48e5-a16d-31694e71a7e2" containerName="payload-pre-processing"
Nov 02 18:53:24.609411 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.609393    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:24.667255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.667213    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"]
Nov 02 18:53:24.704666 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.704635    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvf2\" (UniqueName: \"kubernetes.io/projected/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-kube-api-access-7zvf2\") pod \"payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:24.704854 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.704679    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:24.714331 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.714307    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a0604d6a\""
Nov 02 18:53:24.805478 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.805440    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvf2\" (UniqueName: \"kubernetes.io/projected/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-kube-api-access-7zvf2\") pod \"payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:24.805671 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.805490    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:24.806178 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.806159    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:24.862058 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.861981    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvf2\" (UniqueName: \"kubernetes.io/projected/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-kube-api-access-7zvf2\") pod \"payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:24.922741 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:24.922706    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:25.292366 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:25.292342    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"]
Nov 02 18:53:25.294648 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:53:25.294606    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96cffb2_ebd5_42dd_9525_b61e7a6afec7.slice/crio-d882eefc23f6bb200a2c384abb9310a625334cdc1fd992bd232ef102a010ae58 WatchSource:0}: Error finding container d882eefc23f6bb200a2c384abb9310a625334cdc1fd992bd232ef102a010ae58: Status 404 returned error can't find the container with id d882eefc23f6bb200a2c384abb9310a625334cdc1fd992bd232ef102a010ae58
Nov 02 18:53:25.406073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:25.406034    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5" event={"ID":"f96cffb2-ebd5-42dd-9525-b61e7a6afec7","Type":"ContainerStarted","Data":"9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295"}
Nov 02 18:53:25.406073 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:25.406089    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5" event={"ID":"f96cffb2-ebd5-42dd-9525-b61e7a6afec7","Type":"ContainerStarted","Data":"d882eefc23f6bb200a2c384abb9310a625334cdc1fd992bd232ef102a010ae58"}
Nov 02 18:53:25.406385 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:25.406170    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:25.519837 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:25.519784    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5" podStartSLOduration=1.5197697209999999 podStartE2EDuration="1.519769721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:25.517645665 +0000 UTC m=+5435.774676276" watchObservedRunningTime="2026-11-02 18:53:25.519769721 +0000 UTC m=+5435.776800337"
Nov 02 18:53:26.956290 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:26.956253    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"]
Nov 02 18:53:26.959812 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:26.959793    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.027711 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.027681    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f962ac-8a23-425c-a130-9b788a88a4c5-openshift-service-ca\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.027882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.027726    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77p8t\" (UniqueName: \"kubernetes.io/projected/a7f962ac-8a23-425c-a130-9b788a88a4c5-kube-api-access-77p8t\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.027882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.027839    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.027882 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.027880    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-metrics-certs\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.064597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.064554    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a0604d6a-serving-cert\""
Nov 02 18:53:27.066513 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.066481    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"]
Nov 02 18:53:27.129374 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.129340    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.129575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.129390    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-metrics-certs\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.129575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.129445    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f962ac-8a23-425c-a130-9b788a88a4c5-openshift-service-ca\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.129575 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.129478    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77p8t\" (UniqueName: \"kubernetes.io/projected/a7f962ac-8a23-425c-a130-9b788a88a4c5-kube-api-access-77p8t\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.129575 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:53:27.129502    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-serving-cert: secret "maas-api-e2e-ait-a0604d6a-serving-cert" not found
Nov 02 18:53:27.129575 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:53:27.129573    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls podName:a7f962ac-8a23-425c-a130-9b788a88a4c5 nodeName:}" failed. No retries permitted until 2026-11-02 18:53:27.629557263 +0000 UTC m=+5437.886587884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls") pod "maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf" (UID: "a7f962ac-8a23-425c-a130-9b788a88a4c5") : secret "maas-api-e2e-ait-a0604d6a-serving-cert" not found
Nov 02 18:53:27.130257 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.130240    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f962ac-8a23-425c-a130-9b788a88a4c5-openshift-service-ca\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.132087 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.132070    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-metrics-certs\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.188097 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.188069    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77p8t\" (UniqueName: \"kubernetes.io/projected/a7f962ac-8a23-425c-a130-9b788a88a4c5-kube-api-access-77p8t\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.634874 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.634838    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.637479 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.637449    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls\") pod \"maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:27.872842 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:27.872809    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:28.242939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:28.242914    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"]
Nov 02 18:53:28.244996 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:53:28.244970    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f962ac_8a23_425c_a130_9b788a88a4c5.slice/crio-d3abc49c5b888ca5a8648128b1e788154bc209399c315dd84a885b1af99a39dd WatchSource:0}: Error finding container d3abc49c5b888ca5a8648128b1e788154bc209399c315dd84a885b1af99a39dd: Status 404 returned error can't find the container with id d3abc49c5b888ca5a8648128b1e788154bc209399c315dd84a885b1af99a39dd
Nov 02 18:53:28.246469 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:28.246452    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:53:28.421089 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:28.421056    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf" event={"ID":"a7f962ac-8a23-425c-a130-9b788a88a4c5","Type":"ContainerStarted","Data":"d3abc49c5b888ca5a8648128b1e788154bc209399c315dd84a885b1af99a39dd"}
Nov 02 18:53:29.053562 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:53:29.053521    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:53:29.427680 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:29.427584    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf" event={"ID":"a7f962ac-8a23-425c-a130-9b788a88a4c5","Type":"ContainerStarted","Data":"759a92ae9604b188f220e5036647e56b33b15c75d02f0bf6af1d60f1451fd6fd"}
Nov 02 18:53:29.428186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:29.427764    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:29.548599 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:29.548500    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf" podStartSLOduration=2.600250151 podStartE2EDuration="3.548484838s" totalImagesPullingTime="948.234687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:29.546645775 +0000 UTC m=+5439.803676396" watchObservedRunningTime="2026-11-02 18:53:29.548484838 +0000 UTC m=+5439.805515496"
Nov 02 18:53:35.439672 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:35.439639    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:53:36.414085 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:53:36.414057    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:53:59.052606 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:53:59.052561    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:54:09.130364 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:09.130324    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"]
Nov 02 18:54:14.131397 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.131324    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5" podUID="f96cffb2-ebd5-42dd-9525-b61e7a6afec7" containerName="payload-pre-processing" containerID="cri-o://9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295" gracePeriod=25
Nov 02 18:54:14.383735 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.383665    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:54:14.428424 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.428388    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"]
Nov 02 18:54:14.428616 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.428595    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf" podUID="a7f962ac-8a23-425c-a130-9b788a88a4c5" containerName="maas-api" containerID="cri-o://759a92ae9604b188f220e5036647e56b33b15c75d02f0bf6af1d60f1451fd6fd" gracePeriod=30
Nov 02 18:54:14.452394 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.452365    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-kube-api-access-7zvf2\" (UniqueName: \"kubernetes.io/projected/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-kube-api-access-7zvf2\") pod \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") "
Nov 02 18:54:14.452525 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.452418    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-plugins-config-volume\") pod \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\" (UID: \"f96cffb2-ebd5-42dd-9525-b61e7a6afec7\") "
Nov 02 18:54:14.452844 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.452818    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-plugins-config-volume" pod "f96cffb2-ebd5-42dd-9525-b61e7a6afec7" (UID: "f96cffb2-ebd5-42dd-9525-b61e7a6afec7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:54:14.454962 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.454935    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-kube-api-access-7zvf2" pod "f96cffb2-ebd5-42dd-9525-b61e7a6afec7" (UID: "f96cffb2-ebd5-42dd-9525-b61e7a6afec7"). InnerVolumeSpecName "kube-api-access-7zvf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:54:14.553697 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.553663    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvf2\" (UniqueName: \"kubernetes.io/projected/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-kube-api-access-7zvf2\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:54:14.553835 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.553698    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cffb2-ebd5-42dd-9525-b61e7a6afec7-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:54:14.640251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.640145    4956 generic.go:356] "Generic (PLEG): container finished" podID="a7f962ac-8a23-425c-a130-9b788a88a4c5" containerID="759a92ae9604b188f220e5036647e56b33b15c75d02f0bf6af1d60f1451fd6fd" exitCode=0
Nov 02 18:54:14.640251 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.640231    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf" event={"ID":"a7f962ac-8a23-425c-a130-9b788a88a4c5","Type":"ContainerDied","Data":"759a92ae9604b188f220e5036647e56b33b15c75d02f0bf6af1d60f1451fd6fd"}
Nov 02 18:54:14.641858 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.641831    4956 generic.go:356] "Generic (PLEG): container finished" podID="f96cffb2-ebd5-42dd-9525-b61e7a6afec7" containerID="9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295" exitCode=0
Nov 02 18:54:14.642004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.641917    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5" event={"ID":"f96cffb2-ebd5-42dd-9525-b61e7a6afec7","Type":"ContainerDied","Data":"9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295"}
Nov 02 18:54:14.642004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.641929    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"
Nov 02 18:54:14.642004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.641971    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5" event={"ID":"f96cffb2-ebd5-42dd-9525-b61e7a6afec7","Type":"ContainerDied","Data":"d882eefc23f6bb200a2c384abb9310a625334cdc1fd992bd232ef102a010ae58"}
Nov 02 18:54:14.642004 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.642003    4956 scope.go:122] "RemoveContainer" containerID="9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295"
Nov 02 18:54:14.651844 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.651823    4956 scope.go:122] "RemoveContainer" containerID="9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295"
Nov 02 18:54:14.652197 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:54:14.652178    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295\": container with ID starting with 9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295 not found: ID does not exist" containerID="9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295"
Nov 02 18:54:14.652298 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.652206    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295"} err="failed to get container status \"9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295\": rpc error: code = NotFound desc = could not find container \"9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295\": container with ID starting with 9f22086d0d4aaf05e8fef779c7e0399609f970cc2b03b369f2513073cdaa9295 not found: ID does not exist"
Nov 02 18:54:14.686006 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.685978    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:54:14.755920 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.755865    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f962ac-8a23-425c-a130-9b788a88a4c5-kube-api-access-77p8t\" (UniqueName: \"kubernetes.io/projected/a7f962ac-8a23-425c-a130-9b788a88a4c5-kube-api-access-77p8t\") pod \"a7f962ac-8a23-425c-a130-9b788a88a4c5\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") "
Nov 02 18:54:14.756173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.755943    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls\") pod \"a7f962ac-8a23-425c-a130-9b788a88a4c5\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") "
Nov 02 18:54:14.756173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.756012    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-metrics-certs\") pod \"a7f962ac-8a23-425c-a130-9b788a88a4c5\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") "
Nov 02 18:54:14.756173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.756062    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f962ac-8a23-425c-a130-9b788a88a4c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f962ac-8a23-425c-a130-9b788a88a4c5-openshift-service-ca\") pod \"a7f962ac-8a23-425c-a130-9b788a88a4c5\" (UID: \"a7f962ac-8a23-425c-a130-9b788a88a4c5\") "
Nov 02 18:54:14.756496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.756472    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f962ac-8a23-425c-a130-9b788a88a4c5-openshift-service-ca" pod "a7f962ac-8a23-425c-a130-9b788a88a4c5" (UID: "a7f962ac-8a23-425c-a130-9b788a88a4c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:54:14.758335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.758311    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls" pod "a7f962ac-8a23-425c-a130-9b788a88a4c5" (UID: "a7f962ac-8a23-425c-a130-9b788a88a4c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:54:14.758446 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.758346    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f962ac-8a23-425c-a130-9b788a88a4c5-kube-api-access-77p8t" pod "a7f962ac-8a23-425c-a130-9b788a88a4c5" (UID: "a7f962ac-8a23-425c-a130-9b788a88a4c5"). InnerVolumeSpecName "kube-api-access-77p8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:54:14.758446 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.758345    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-metrics-certs" pod "a7f962ac-8a23-425c-a130-9b788a88a4c5" (UID: "a7f962ac-8a23-425c-a130-9b788a88a4c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:54:14.812135 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.812084    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"]
Nov 02 18:54:14.815336 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.815306    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0604d6a-7667db56f5-8jnj5"]
Nov 02 18:54:14.857579 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.857545    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:54:14.857579 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.857577    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f962ac-8a23-425c-a130-9b788a88a4c5-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:54:14.857774 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.857591    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77p8t\" (UniqueName: \"kubernetes.io/projected/a7f962ac-8a23-425c-a130-9b788a88a4c5-kube-api-access-77p8t\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:54:14.857774 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:14.857606    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f962ac-8a23-425c-a130-9b788a88a4c5-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:54:15.647255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:15.647229    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"
Nov 02 18:54:15.647743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:15.647252    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf" event={"ID":"a7f962ac-8a23-425c-a130-9b788a88a4c5","Type":"ContainerDied","Data":"d3abc49c5b888ca5a8648128b1e788154bc209399c315dd84a885b1af99a39dd"}
Nov 02 18:54:15.647743 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:15.647346    4956 scope.go:122] "RemoveContainer" containerID="759a92ae9604b188f220e5036647e56b33b15c75d02f0bf6af1d60f1451fd6fd"
Nov 02 18:54:15.809713 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:15.809676    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"]
Nov 02 18:54:15.864387 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:15.864349    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96cffb2-ebd5-42dd-9525-b61e7a6afec7" path="/var/lib/kubelet/pods/f96cffb2-ebd5-42dd-9525-b61e7a6afec7/volumes"
Nov 02 18:54:15.864646 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:15.864632    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a0604d6a-7f4c947746-qz2wf"]
Nov 02 18:54:17.864256 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:17.864225    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f962ac-8a23-425c-a130-9b788a88a4c5" path="/var/lib/kubelet/pods/a7f962ac-8a23-425c-a130-9b788a88a4c5/volumes"
Nov 02 18:54:29.054775 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:54:29.054735    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:54:53.010958 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.010929    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"]
Nov 02 18:54:53.011400 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.011315    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a7f962ac-8a23-425c-a130-9b788a88a4c5" containerName="maas-api"
Nov 02 18:54:53.011400 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.011326    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f962ac-8a23-425c-a130-9b788a88a4c5" containerName="maas-api"
Nov 02 18:54:53.011400 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.011354    4956 cpu_manager.go:414] "containerMap: removing container" podUID="f96cffb2-ebd5-42dd-9525-b61e7a6afec7" containerName="payload-pre-processing"
Nov 02 18:54:53.011400 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.011360    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96cffb2-ebd5-42dd-9525-b61e7a6afec7" containerName="payload-pre-processing"
Nov 02 18:54:53.011542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.011424    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96cffb2-ebd5-42dd-9525-b61e7a6afec7" containerName="payload-pre-processing"
Nov 02 18:54:53.011542 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.011435    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f962ac-8a23-425c-a130-9b788a88a4c5" containerName="maas-api"
Nov 02 18:54:53.015836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.015817    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.067492 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.067464    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\""
Nov 02 18:54:53.067682 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.067628    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 18:54:53.107767 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.107733    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d345262e-c751-4322-afe4-579967efeaf4-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.107937 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.107791    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d345262e-c751-4322-afe4-579967efeaf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.107937 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.107836    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.107937 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.107883    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.108079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.107963    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.108079 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.108023    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.108191 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.108081    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmlb\" (UniqueName: \"kubernetes.io/projected/d345262e-c751-4322-afe4-579967efeaf4-kube-api-access-cfmlb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.108191 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.108165    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.108283 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.108196    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d345262e-c751-4322-afe4-579967efeaf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.118768 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.118741    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"]
Nov 02 18:54:53.209410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209373    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmlb\" (UniqueName: \"kubernetes.io/projected/d345262e-c751-4322-afe4-579967efeaf4-kube-api-access-cfmlb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.209410 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209412    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.209699 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209439    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d345262e-c751-4322-afe4-579967efeaf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.209699 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209475    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d345262e-c751-4322-afe4-579967efeaf4-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.209699 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209518    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d345262e-c751-4322-afe4-579967efeaf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.209699 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209654    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209717    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209777    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209824    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210024 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.209834    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.210107    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.210179    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.210250    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d345262e-c751-4322-afe4-579967efeaf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.210431 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.210311    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.212029 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.212004    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d345262e-c751-4322-afe4-579967efeaf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.212179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.212140    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d345262e-c751-4322-afe4-579967efeaf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.266203 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.266115    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d345262e-c751-4322-afe4-579967efeaf4-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.266340 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.266277    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmlb\" (UniqueName: \"kubernetes.io/projected/d345262e-c751-4322-afe4-579967efeaf4-kube-api-access-cfmlb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tfl27\" (UID: \"d345262e-c751-4322-afe4-579967efeaf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.329000 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.328966    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:53.749823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.749796    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"]
Nov 02 18:54:53.752457 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:54:53.752432    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd345262e_c751_4322_afe4_579967efeaf4.slice/crio-e7f973337b349f0535be80fab913ed418174409275a1d53a8943e881d4a972d1 WatchSource:0}: Error finding container e7f973337b349f0535be80fab913ed418174409275a1d53a8943e881d4a972d1: Status 404 returned error can't find the container with id e7f973337b349f0535be80fab913ed418174409275a1d53a8943e881d4a972d1
Nov 02 18:54:53.754539 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.754507    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:54:53.754608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.754571    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:54:53.754608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.754597    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:54:53.822992 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.822955    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" event={"ID":"d345262e-c751-4322-afe4-579967efeaf4","Type":"ContainerStarted","Data":"0a7bf445fdf090ffce8b509cf95b5325d20f5ee3533b6eb11a1e3d14c6e78b8c"}
Nov 02 18:54:53.823163 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.823018    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" event={"ID":"d345262e-c751-4322-afe4-579967efeaf4","Type":"ContainerStarted","Data":"e7f973337b349f0535be80fab913ed418174409275a1d53a8943e881d4a972d1"}
Nov 02 18:54:53.938989 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:53.938936    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podStartSLOduration=1.938921073 podStartE2EDuration="1.938921073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:54:53.937425247 +0000 UTC m=+5524.194455868" watchObservedRunningTime="2026-11-02 18:54:53.938921073 +0000 UTC m=+5524.195951685"
Nov 02 18:54:54.330865 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:54.330818    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:54:54.331578 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:54.331233    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:54:54.331793 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:54.331765    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:54:55.334147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:55.332539    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:54:55.334147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:55.332605    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:54:56.342146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:56.338537    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:54:56.342146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:56.338604    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:54:57.334145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:57.329484    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:54:57.334145 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:57.329547    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:54:58.332607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:58.332566    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:54:58.333268 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:58.333236    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:54:59.073864 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:54:59.073829    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:54:59.336208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:59.329487    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:54:59.336208 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:54:59.329550    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:55:00.330224 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:00.330187    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:55:00.330443 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:00.330249    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:55:01.330179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:01.330144    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 18:55:01.330581 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:01.330203    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 18:55:02.342482 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:02.342444    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:55:02.342959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:02.342763    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:55:02.344900 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:02.344876    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27"
Nov 02 18:55:08.124999 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.124968    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"]
Nov 02 18:55:08.128636 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.128619    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168402 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168374    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168580 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168411    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168580 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168436    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168580 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168474    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168580 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168556    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168580 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168575    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxgjs\" (UniqueName: \"kubernetes.io/projected/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-kube-api-access-lxgjs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168598    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168619    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.168794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.168701    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.232054 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.232021    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"]
Nov 02 18:55:08.269632 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.269596    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.269632 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.269636    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.269901 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.269667    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.269901 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.269789    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.269901 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.269877    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.269909    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxgjs\" (UniqueName: \"kubernetes.io/projected/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-kube-api-access-lxgjs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.269958    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270091 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.270051    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.270155    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.270184    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.270186    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270292 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.270244    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.270472    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.270536 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.270515    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.272390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.272364    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.272634 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.272617    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.325939 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.325897    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.326171 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.326150    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxgjs\" (UniqueName: \"kubernetes.io/projected/0b8ee4be-5295-401a-aa31-dc9cd699e5b6-kube-api-access-lxgjs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-j8pgt\" (UID: \"0b8ee4be-5295-401a-aa31-dc9cd699e5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.442534 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.442439    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:08.864255 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.864236    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"]
Nov 02 18:55:08.866619 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:55:08.866587    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8ee4be_5295_401a_aa31_dc9cd699e5b6.slice/crio-706067b69ec3045c6d67b9b3ca1ea8f4b12260f996edb2383efb224dba919706 WatchSource:0}: Error finding container 706067b69ec3045c6d67b9b3ca1ea8f4b12260f996edb2383efb224dba919706: Status 404 returned error can't find the container with id 706067b69ec3045c6d67b9b3ca1ea8f4b12260f996edb2383efb224dba919706
Nov 02 18:55:08.868649 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.868615    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:55:08.868713 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.868682    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:55:08.868755 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.868710    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:55:08.912391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:08.912361    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" event={"ID":"0b8ee4be-5295-401a-aa31-dc9cd699e5b6","Type":"ContainerStarted","Data":"706067b69ec3045c6d67b9b3ca1ea8f4b12260f996edb2383efb224dba919706"}
Nov 02 18:55:09.918390 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:09.918343    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" event={"ID":"0b8ee4be-5295-401a-aa31-dc9cd699e5b6","Type":"ContainerStarted","Data":"3b26ccdd010674a791cca512e4d0927d69bded01de0853cd650010f6e46b0d6d"}
Nov 02 18:55:10.147147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:10.144240    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podStartSLOduration=2.144218274 podStartE2EDuration="2.144218274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:10.081781337 +0000 UTC m=+5540.338811957" watchObservedRunningTime="2026-11-02 18:55:10.144218274 +0000 UTC m=+5540.401248897"
Nov 02 18:55:10.450337 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:10.450300    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:10.451823 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:10.451800    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:55:10.451877 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:10.451847    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 18:55:11.450602 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:11.450562    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:55:11.451000 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:11.450621    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 18:55:12.444146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:12.443517    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:55:12.444146 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:12.443581    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 18:55:13.447147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:13.445596    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:55:13.447147 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:13.445661    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 18:55:14.446155 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:14.445576    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:55:14.446155 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:14.445639    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 18:55:15.443456 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:15.443409    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:55:15.443996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:15.443478    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 18:55:16.443405 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:16.443371    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:55:16.443645 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:16.443435    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 18:55:17.447302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:17.447272    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:17.447780 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:17.447689    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:17.448511 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:17.448494    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt"
Nov 02 18:55:29.051835 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:55:29.051773    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:55:53.467195 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.467156    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"]
Nov 02 18:55:53.470895 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.470874    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509540 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509503    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788c652-a093-414f-bcf3-c96c544ef908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509540 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509551    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0788c652-a093-414f-bcf3-c96c544ef908-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509632    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcp8v\" (UniqueName: \"kubernetes.io/projected/0788c652-a093-414f-bcf3-c96c544ef908-kube-api-access-jcp8v\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509711    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509750    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788c652-a093-414f-bcf3-c96c544ef908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509766    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509802    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.509830 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509819    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.510235 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.509974    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.610794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.610749    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.610794 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.610799    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.610870    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.610907    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788c652-a093-414f-bcf3-c96c544ef908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.610952    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0788c652-a093-414f-bcf3-c96c544ef908-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.610983    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcp8v\" (UniqueName: \"kubernetes.io/projected/0788c652-a093-414f-bcf3-c96c544ef908-kube-api-access-jcp8v\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611050 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.611016    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611357 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.611066    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788c652-a093-414f-bcf3-c96c544ef908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611357 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.611094    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611357 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.611278    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611357 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.611336    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611590 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.611446    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.611807 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.611777    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.612195 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.612169    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788c652-a093-414f-bcf3-c96c544ef908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.614104 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.614065    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788c652-a093-414f-bcf3-c96c544ef908-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.614819 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.614798    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788c652-a093-414f-bcf3-c96c544ef908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.672685 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.672617    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0788c652-a093-414f-bcf3-c96c544ef908-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.680005 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.679975    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"]
Nov 02 18:55:53.684687 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.684611    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcp8v\" (UniqueName: \"kubernetes.io/projected/0788c652-a093-414f-bcf3-c96c544ef908-kube-api-access-jcp8v\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-268pz\" (UID: \"0788c652-a093-414f-bcf3-c96c544ef908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:53.784761 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:53.784658    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:54.273143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:54.269622    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"]
Nov 02 18:55:54.279653 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:54.279611    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:55:54.286373 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:54.286325    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:55:54.286613 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:54.286586    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:55:55.163150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:55.162388    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" event={"ID":"0788c652-a093-414f-bcf3-c96c544ef908","Type":"ContainerStarted","Data":"4e1ed68ce33356814a00d0b9b2451ad3e5a624bc6d790415f36ba51ac41d2a7a"}
Nov 02 18:55:55.163150 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:55.162460    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" event={"ID":"0788c652-a093-414f-bcf3-c96c544ef908","Type":"ContainerStarted","Data":"30d70508560be94d00a0aee68750c1f07326ebb0f03a00e0ae561c1c2e539e35"}
Nov 02 18:55:55.349399 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:55.349330    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podStartSLOduration=2.349310936 podStartE2EDuration="2.349310936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:55.349039664 +0000 UTC m=+5585.606070284" watchObservedRunningTime="2026-11-02 18:55:55.349310936 +0000 UTC m=+5585.606341554"
Nov 02 18:55:55.785447 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:55.785403    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:55:55.786986 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:55.786962    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 18:55:55.787139 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:55.787031    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 18:55:56.785594 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:56.785557    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 18:55:56.785991 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:56.785614    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 18:55:57.785761 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:57.785724    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 18:55:57.786218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:57.785783    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 18:55:58.786504 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:58.786467    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 18:55:58.786902 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:58.786564    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 18:55:59.074454 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:55:59.074370    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:55:59.786244 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:59.786206    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 18:55:59.786445 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:55:59.786266    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 18:56:00.785829 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:00.785787    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 18:56:00.786280 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:00.785849    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 18:56:01.786248 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:01.786204    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 18:56:01.786695 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:01.786262    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 18:56:03.788148 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:03.787720    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:56:03.788676 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:03.787832    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:56:04.832342 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:04.832300    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:56:04.832975 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:04.832940    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:56:05.587533 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:05.587493    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:56:05.587856 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:05.587831    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:56:05.588547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:05.588528    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz"
Nov 02 18:56:29.051563 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:56:29.051520    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:56:55.104173 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.104136    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"]
Nov 02 18:56:55.107962 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.107937    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.210318 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.210287    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4397aa02-kserve-self-signed-certs\""
Nov 02 18:56:55.212082 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.212062    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"]
Nov 02 18:56:55.296797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.296758    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6dx\" (UniqueName: \"kubernetes.io/projected/b2b267dc-e835-472b-9f5c-7443928084bd-kube-api-access-2k6dx\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.296988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.296812    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b267dc-e835-472b-9f5c-7443928084bd-tls-certs\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.296988 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.296869    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-dshm\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.297157 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.297011    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-home\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.297157 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.297054    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-tmp-dir\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.297157 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.297091    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-model-cache\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.398592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.398494    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b267dc-e835-472b-9f5c-7443928084bd-tls-certs\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.398592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.398538    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-dshm\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.398592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.398582    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-home\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.398904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.398606    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-tmp-dir\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.398904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.398633    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-model-cache\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.398904 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.398701    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6dx\" (UniqueName: \"kubernetes.io/projected/b2b267dc-e835-472b-9f5c-7443928084bd-kube-api-access-2k6dx\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.399166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.399139    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-model-cache\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.399245 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.399165    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-tmp-dir\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.399308 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.399244    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-home\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.401017 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.400994    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-dshm\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.401438 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.401416    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b267dc-e835-472b-9f5c-7443928084bd-tls-certs\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.456502 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.456466    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6dx\" (UniqueName: \"kubernetes.io/projected/b2b267dc-e835-472b-9f5c-7443928084bd-kube-api-access-2k6dx\") pod \"e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") " pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:55.719768 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:55.719729    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:56.108054 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:56.107984    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"]
Nov 02 18:56:56.110522 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:56:56.110486    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2b267dc_e835_472b_9f5c_7443928084bd.slice/crio-20bf88fe0ba10a36b1f8be41cce3cd2725ff581189580bb390575a2b7389064c WatchSource:0}: Error finding container 20bf88fe0ba10a36b1f8be41cce3cd2725ff581189580bb390575a2b7389064c: Status 404 returned error can't find the container with id 20bf88fe0ba10a36b1f8be41cce3cd2725ff581189580bb390575a2b7389064c
Nov 02 18:56:56.462197 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:56.462161    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m" event={"ID":"b2b267dc-e835-472b-9f5c-7443928084bd","Type":"ContainerStarted","Data":"edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76"}
Nov 02 18:56:56.462391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:56.462210    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m" event={"ID":"b2b267dc-e835-472b-9f5c-7443928084bd","Type":"ContainerStarted","Data":"20bf88fe0ba10a36b1f8be41cce3cd2725ff581189580bb390575a2b7389064c"}
Nov 02 18:56:56.576817 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:56.576758    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m" podStartSLOduration=1.576739846 podStartE2EDuration="1.576739846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:56.576315072 +0000 UTC m=+5646.833345691" watchObservedRunningTime="2026-11-02 18:56:56.576739846 +0000 UTC m=+5646.833770469"
Nov 02 18:56:56.719959 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:56:56.719862    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:56:59.052551 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:56:59.052510    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:57:05.719940 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:57:05.719897    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:57:05.733375 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:57:05.733345    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:57:06.540037 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:57:06.540002    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:57:29.052558 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:57:29.052518    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:57:50.688746 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:57:50.688720    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:57:50.697435 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:57:50.697412    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 18:57:59.051603 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:57:59.051569    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:58:03.832649 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:03.832561    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"]
Nov 02 18:58:18.853822 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:18.853758    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m" podUID="b2b267dc-e835-472b-9f5c-7443928084bd" containerName="main" containerID="cri-o://edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76" gracePeriod=45
Nov 02 18:58:19.114853 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.114792    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:58:19.178302 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178261    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-tmp-dir\") pod \"b2b267dc-e835-472b-9f5c-7443928084bd\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") "
Nov 02 18:58:19.178485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178318    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b267dc-e835-472b-9f5c-7443928084bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b267dc-e835-472b-9f5c-7443928084bd-tls-certs\") pod \"b2b267dc-e835-472b-9f5c-7443928084bd\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") "
Nov 02 18:58:19.178485 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178353    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-model-cache\") pod \"b2b267dc-e835-472b-9f5c-7443928084bd\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") "
Nov 02 18:58:19.178607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178479    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b267dc-e835-472b-9f5c-7443928084bd-kube-api-access-2k6dx\" (UniqueName: \"kubernetes.io/projected/b2b267dc-e835-472b-9f5c-7443928084bd-kube-api-access-2k6dx\") pod \"b2b267dc-e835-472b-9f5c-7443928084bd\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") "
Nov 02 18:58:19.178607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178532    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-dshm\") pod \"b2b267dc-e835-472b-9f5c-7443928084bd\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") "
Nov 02 18:58:19.178607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178527    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-tmp-dir" pod "b2b267dc-e835-472b-9f5c-7443928084bd" (UID: "b2b267dc-e835-472b-9f5c-7443928084bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:58:19.178607 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178566    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-home\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-home\") pod \"b2b267dc-e835-472b-9f5c-7443928084bd\" (UID: \"b2b267dc-e835-472b-9f5c-7443928084bd\") "
Nov 02 18:58:19.178836 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178618    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-model-cache" pod "b2b267dc-e835-472b-9f5c-7443928084bd" (UID: "b2b267dc-e835-472b-9f5c-7443928084bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:58:19.178897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178831    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-home" pod "b2b267dc-e835-472b-9f5c-7443928084bd" (UID: "b2b267dc-e835-472b-9f5c-7443928084bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:58:19.178897 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178882    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:19.178972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.178899    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:19.180808 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.180784    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-dshm" pod "b2b267dc-e835-472b-9f5c-7443928084bd" (UID: "b2b267dc-e835-472b-9f5c-7443928084bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:58:19.180943 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.180849    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b267dc-e835-472b-9f5c-7443928084bd-tls-certs" pod "b2b267dc-e835-472b-9f5c-7443928084bd" (UID: "b2b267dc-e835-472b-9f5c-7443928084bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:58:19.180943 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.180860    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b267dc-e835-472b-9f5c-7443928084bd-kube-api-access-2k6dx" pod "b2b267dc-e835-472b-9f5c-7443928084bd" (UID: "b2b267dc-e835-472b-9f5c-7443928084bd"). InnerVolumeSpecName "kube-api-access-2k6dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:58:19.280334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.280294    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k6dx\" (UniqueName: \"kubernetes.io/projected/b2b267dc-e835-472b-9f5c-7443928084bd-kube-api-access-2k6dx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:19.280334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.280327    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:19.280334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.280341    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b267dc-e835-472b-9f5c-7443928084bd-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:19.280608 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.280352    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b267dc-e835-472b-9f5c-7443928084bd-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:19.896815 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.896781    4956 generic.go:356] "Generic (PLEG): container finished" podID="b2b267dc-e835-472b-9f5c-7443928084bd" containerID="edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76" exitCode=0
Nov 02 18:58:19.896815 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.896818    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"
Nov 02 18:58:19.897335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.896858    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m" event={"ID":"b2b267dc-e835-472b-9f5c-7443928084bd","Type":"ContainerDied","Data":"edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76"}
Nov 02 18:58:19.897335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.896904    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m" event={"ID":"b2b267dc-e835-472b-9f5c-7443928084bd","Type":"ContainerDied","Data":"20bf88fe0ba10a36b1f8be41cce3cd2725ff581189580bb390575a2b7389064c"}
Nov 02 18:58:19.897335 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.896928    4956 scope.go:122] "RemoveContainer" containerID="edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76"
Nov 02 18:58:19.905592 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.905572    4956 scope.go:122] "RemoveContainer" containerID="edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76"
Nov 02 18:58:19.905827 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:58:19.905809    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76\": container with ID starting with edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76 not found: ID does not exist" containerID="edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76"
Nov 02 18:58:19.905893 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:19.905836    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76"} err="failed to get container status \"edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76\": rpc error: code = NotFound desc = could not find container \"edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76\": container with ID starting with edca0615a56eb26b3b5744f4b5d88d51fed018c056eb51509523146671092e76 not found: ID does not exist"
Nov 02 18:58:20.058818 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:20.058781    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"]
Nov 02 18:58:20.108186 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:20.108153    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-56cd547bfd-6nr4m"]
Nov 02 18:58:21.863720 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:21.863685    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b267dc-e835-472b-9f5c-7443928084bd" path="/var/lib/kubelet/pods/b2b267dc-e835-472b-9f5c-7443928084bd/volumes"
Nov 02 18:58:22.435638 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.435607    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"]
Nov 02 18:58:22.436041 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.436028    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b2b267dc-e835-472b-9f5c-7443928084bd" containerName="main"
Nov 02 18:58:22.436086 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.436044    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b267dc-e835-472b-9f5c-7443928084bd" containerName="main"
Nov 02 18:58:22.436166 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.436156    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b267dc-e835-472b-9f5c-7443928084bd" containerName="main"
Nov 02 18:58:22.440637 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.440616    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.492572 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.492541    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-6edebe-openshift-default-dockercfg-fdv4f\""
Nov 02 18:58:22.511193 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511162    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-certs\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511222    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-credential-socket\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511377 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511337    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-socket\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511412    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-data\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511496 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511454    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddeb41e6-55ea-4267-9af1-137293b006fb-istiod-ca-cert\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511508    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-token\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511591 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511556    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwtj\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-kube-api-access-mwwtj\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511676 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511586    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-podinfo\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.511676 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.511618    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-envoy\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.544131 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.544087    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"]
Nov 02 18:58:22.612644 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612581    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddeb41e6-55ea-4267-9af1-137293b006fb-istiod-ca-cert\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.612843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612674    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-token\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.612843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612702    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwtj\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-kube-api-access-mwwtj\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.612843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612736    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-podinfo\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.612843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612766    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-envoy\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.612843 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612811    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-certs\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612875    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-credential-socket\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612929    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-socket\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613101 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.612961    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-data\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613369 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.613345    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-credential-socket\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.613378    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-certs\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613451 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.613395    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-data\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613550 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.613464    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddeb41e6-55ea-4267-9af1-137293b006fb-istiod-ca-cert\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.613550 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.613485    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-socket\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.615378 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.615344    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-envoy\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.615500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.615459    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-podinfo\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.670373 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.670328    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-token\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.670763 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.670728    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwtj\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-kube-api-access-mwwtj\") pod \"e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") " pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:22.753034 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:22.752987    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:23.179044 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:23.179013    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"]
Nov 02 18:58:23.180845 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:58:23.180818    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeb41e6_55ea_4267_9af1_137293b006fb.slice/crio-b5447dce054c3da366f972d731d005e51b70efb87537be8aaf90ce14ca59c6e0 WatchSource:0}: Error finding container b5447dce054c3da366f972d731d005e51b70efb87537be8aaf90ce14ca59c6e0: Status 404 returned error can't find the container with id b5447dce054c3da366f972d731d005e51b70efb87537be8aaf90ce14ca59c6e0
Nov 02 18:58:23.182968 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:23.182941    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:58:23.183056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:23.182999    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:58:23.183056 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:23.183030    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:58:23.917369 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:23.917330    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" event={"ID":"ddeb41e6-55ea-4267-9af1-137293b006fb","Type":"ContainerStarted","Data":"c7db7bd4206d71ba8ea086d51c121f6f826a67d2c4ce16907330f0c84457042e"}
Nov 02 18:58:23.917557 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:23.917384    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" event={"ID":"ddeb41e6-55ea-4267-9af1-137293b006fb","Type":"ContainerStarted","Data":"b5447dce054c3da366f972d731d005e51b70efb87537be8aaf90ce14ca59c6e0"}
Nov 02 18:58:24.054449 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:24.054392    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podStartSLOduration=2.054377903 podStartE2EDuration="2.054377903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:58:24.052407331 +0000 UTC m=+5734.309437950" watchObservedRunningTime="2026-11-02 18:58:24.054377903 +0000 UTC m=+5734.311408516"
Nov 02 18:58:24.753748 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:24.753713    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:24.759016 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:24.758988    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:24.922075 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:24.922042    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:24.923141 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:24.923108    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:29.052826 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:58:29.052794    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:58:30.861218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:30.861184    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"]
Nov 02 18:58:30.864947 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:30.864932    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:30.920861 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:30.920827    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"]
Nov 02 18:58:30.945334 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:30.945247    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9f8b\" (UniqueName: \"kubernetes.io/projected/bfddb85b-cecf-41b0-a648-b34a650ff227-kube-api-access-w9f8b\") pod \"payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:30.945547 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:30.945377    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfddb85b-cecf-41b0-a648-b34a650ff227-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:30.968135 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:30.968080    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6edebe\""
Nov 02 18:58:30.968135 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:30.968080    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-409fa606-dockercfg-x5kzx\""
Nov 02 18:58:31.024136 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.024092    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"]
Nov 02 18:58:31.027792 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.027768    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.046009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.045979    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfddb85b-cecf-41b0-a648-b34a650ff227-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:31.046188 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.046088    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9f8b\" (UniqueName: \"kubernetes.io/projected/bfddb85b-cecf-41b0-a648-b34a650ff227-kube-api-access-w9f8b\") pod \"payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:31.046677 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.046658    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfddb85b-cecf-41b0-a648-b34a650ff227-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:31.087093 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.087060    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"]
Nov 02 18:58:31.107681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.107645    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9f8b\" (UniqueName: \"kubernetes.io/projected/bfddb85b-cecf-41b0-a648-b34a650ff227-kube-api-access-w9f8b\") pod \"payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:31.147175 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.147068    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4xz\" (UniqueName: \"kubernetes.io/projected/b9ce716a-b3c5-4655-8712-b5b593da3da0-kube-api-access-fq4xz\") pod \"payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.147356 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.147224    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ce716a-b3c5-4655-8712-b5b593da3da0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.178855 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.178823    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:31.248495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.248078    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4xz\" (UniqueName: \"kubernetes.io/projected/b9ce716a-b3c5-4655-8712-b5b593da3da0-kube-api-access-fq4xz\") pod \"payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.248495 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.248194    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ce716a-b3c5-4655-8712-b5b593da3da0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.249188 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.249157    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ce716a-b3c5-4655-8712-b5b593da3da0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.311178 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.311137    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4xz\" (UniqueName: \"kubernetes.io/projected/b9ce716a-b3c5-4655-8712-b5b593da3da0-kube-api-access-fq4xz\") pod \"payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.341394 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.341349    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.552797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.552768    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"]
Nov 02 18:58:31.554458 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:58:31.554409    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfddb85b_cecf_41b0_a648_b34a650ff227.slice/crio-e3bba2332f589e81a5842760d709b03b91e658e6834be9a349e18136af388937 WatchSource:0}: Error finding container e3bba2332f589e81a5842760d709b03b91e658e6834be9a349e18136af388937: Status 404 returned error can't find the container with id e3bba2332f589e81a5842760d709b03b91e658e6834be9a349e18136af388937
Nov 02 18:58:31.714151 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.714109    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"]
Nov 02 18:58:31.716659 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:58:31.716635    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ce716a_b3c5_4655_8712_b5b593da3da0.slice/crio-4609bf5b606ae94e3b330f30fa8269eaa354fafa34dff127e40acb6c04c1dd05 WatchSource:0}: Error finding container 4609bf5b606ae94e3b330f30fa8269eaa354fafa34dff127e40acb6c04c1dd05: Status 404 returned error can't find the container with id 4609bf5b606ae94e3b330f30fa8269eaa354fafa34dff127e40acb6c04c1dd05
Nov 02 18:58:31.962736 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.962700    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" event={"ID":"bfddb85b-cecf-41b0-a648-b34a650ff227","Type":"ContainerStarted","Data":"0f31f73e2b87bb79a0a47b7935df3d76c322ba9dcb2f9176f4ab096ad8fbc912"}
Nov 02 18:58:31.963289 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.962774    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:31.963289 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.962898    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" event={"ID":"bfddb85b-cecf-41b0-a648-b34a650ff227","Type":"ContainerStarted","Data":"e3bba2332f589e81a5842760d709b03b91e658e6834be9a349e18136af388937"}
Nov 02 18:58:31.964729 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.964701    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" event={"ID":"b9ce716a-b3c5-4655-8712-b5b593da3da0","Type":"ContainerStarted","Data":"954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a"}
Nov 02 18:58:31.964887 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.964757    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:31.964887 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:31.964784    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" event={"ID":"b9ce716a-b3c5-4655-8712-b5b593da3da0","Type":"ContainerStarted","Data":"4609bf5b606ae94e3b330f30fa8269eaa354fafa34dff127e40acb6c04c1dd05"}
Nov 02 18:58:32.076179 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:32.076104    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" podStartSLOduration=2.076088123 podStartE2EDuration="2.076088123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:58:32.074728418 +0000 UTC m=+5742.331759039" watchObservedRunningTime="2026-11-02 18:58:32.076088123 +0000 UTC m=+5742.333118742"
Nov 02 18:58:32.187281 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:32.187227    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" podStartSLOduration=2.187211414 podStartE2EDuration="2.187211414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:58:32.186826171 +0000 UTC m=+5742.443856794" watchObservedRunningTime="2026-11-02 18:58:32.187211414 +0000 UTC m=+5742.444242037"
Nov 02 18:58:33.261068 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.261033    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"]
Nov 02 18:58:33.264723 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.264702    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.316573 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.316546    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6edebe-serving-cert\""
Nov 02 18:58:33.368524 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.368490    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"]
Nov 02 18:58:33.369745 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.369711    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.369941 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.369915    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8cn\" (UniqueName: \"kubernetes.io/projected/1b47e4da-8d86-482c-95e0-69edb3143162-kube-api-access-xj8cn\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.369996 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.369981    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.370072 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.370051    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b47e4da-8d86-482c-95e0-69edb3143162-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.471432 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.471393    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8cn\" (UniqueName: \"kubernetes.io/projected/1b47e4da-8d86-482c-95e0-69edb3143162-kube-api-access-xj8cn\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.471635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.471442    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.471635 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:58:33.471558    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-serving-cert: secret "maas-api-e2e-worker-single-main-6edebe-serving-cert" not found
Nov 02 18:58:33.471635 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.471573    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b47e4da-8d86-482c-95e0-69edb3143162-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.471635 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:58:33.471621    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls podName:1b47e4da-8d86-482c-95e0-69edb3143162 nodeName:}" failed. No retries permitted until 2026-11-02 18:58:33.971600038 +0000 UTC m=+5744.228630655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls") pod "maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4" (UID: "1b47e4da-8d86-482c-95e0-69edb3143162") : secret "maas-api-e2e-worker-single-main-6edebe-serving-cert" not found
Nov 02 18:58:33.471783 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.471692    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.472234 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.472208    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b47e4da-8d86-482c-95e0-69edb3143162-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.474190 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.474169    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.528787 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.528706    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8cn\" (UniqueName: \"kubernetes.io/projected/1b47e4da-8d86-482c-95e0-69edb3143162-kube-api-access-xj8cn\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.977142 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.977089    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:33.979633 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:33.979604    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:34.178599 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:34.178561    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:34.553681 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:34.553656    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"]
Nov 02 18:58:34.555690 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:58:34.555650    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b47e4da_8d86_482c_95e0_69edb3143162.slice/crio-2ba5fd7a0960016fe339ace9a677a2615134ad8ec68f1fb4e6a742f1e30b77b9 WatchSource:0}: Error finding container 2ba5fd7a0960016fe339ace9a677a2615134ad8ec68f1fb4e6a742f1e30b77b9: Status 404 returned error can't find the container with id 2ba5fd7a0960016fe339ace9a677a2615134ad8ec68f1fb4e6a742f1e30b77b9
Nov 02 18:58:34.557245 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:34.557227    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:58:34.980106 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:34.980066    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4" event={"ID":"1b47e4da-8d86-482c-95e0-69edb3143162","Type":"ContainerStarted","Data":"2ba5fd7a0960016fe339ace9a677a2615134ad8ec68f1fb4e6a742f1e30b77b9"}
Nov 02 18:58:35.987009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:35.986975    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4" event={"ID":"1b47e4da-8d86-482c-95e0-69edb3143162","Type":"ContainerStarted","Data":"391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d"}
Nov 02 18:58:35.987506 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:35.987080    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:36.100740 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:36.100683    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4" podStartSLOduration=2.194316273 podStartE2EDuration="3.100668712s" totalImagesPullingTime="906.352439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:58:36.100425621 +0000 UTC m=+5746.357456240" watchObservedRunningTime="2026-11-02 18:58:36.100668712 +0000 UTC m=+5746.357699330"
Nov 02 18:58:41.997738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:41.997691    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 18:58:42.971738 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:42.971710    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 18:58:42.971928 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:42.971862    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 18:58:50.079535 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:50.079430    4956 generic.go:356] "Generic (PLEG): container finished" podID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerID="c7db7bd4206d71ba8ea086d51c121f6f826a67d2c4ce16907330f0c84457042e" exitCode=0
Nov 02 18:58:50.079535 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:50.079519    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" event={"ID":"ddeb41e6-55ea-4267-9af1-137293b006fb","Type":"ContainerDied","Data":"c7db7bd4206d71ba8ea086d51c121f6f826a67d2c4ce16907330f0c84457042e"}
Nov 02 18:58:50.080009 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:50.079895    4956 scope.go:122] "RemoveContainer" containerID="c7db7bd4206d71ba8ea086d51c121f6f826a67d2c4ce16907330f0c84457042e"
Nov 02 18:58:50.081059 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:50.081032    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:58:50.081167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:50.081094    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:58:50.081167 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:50.081156    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:58:51.087415 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:51.087382    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" event={"ID":"ddeb41e6-55ea-4267-9af1-137293b006fb","Type":"ContainerStarted","Data":"8b03b7de2bb4957f2c967e1fe0386cfe14cc0cc74f146fc895a10f24b79302b5"}
Nov 02 18:58:51.753382 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:51.753353    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:51.754896 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:51.754856    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 18:58:51.755048 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:51.754919    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 18:58:52.753927 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:52.753886    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 18:58:52.754380 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:52.753958    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 18:58:53.754375 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:53.754327    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 18:58:53.754808 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:53.754423    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 18:58:54.753972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:54.753937    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 18:58:54.754218 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:54.754003    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 18:58:54.922316 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:54.922289    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:55.358597 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.357925    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"]
Nov 02 18:58:55.362132 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.362088    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.393878 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.393841    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"]
Nov 02 18:58:55.448143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.447997    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6edebe\"/\"openshift-service-ca.crt\""
Nov 02 18:58:55.448143 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.448086    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6edebe\"/\"kube-root-ca.crt\""
Nov 02 18:58:55.448423 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.448159    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6edebe\"/\"default-dockercfg-27rzm\""
Nov 02 18:58:55.463875 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.463842    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6edebe\"/\"simulator-single-main-6edebe-kserve-self-signed-certs\""
Nov 02 18:58:55.508391 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.508346    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-tmp-dir\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.508611 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.508415    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-dshm\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.508683 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.508602    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9ghs\" (UniqueName: \"kubernetes.io/projected/38add8ee-b4fb-4927-bca8-31103bceb5bc-kube-api-access-s9ghs\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.508683 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.508665    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38add8ee-b4fb-4927-bca8-31103bceb5bc-tls-certs\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.508760 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.508731    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-home\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.508797 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.508757    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-model-cache\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610043 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.609954    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-tmp-dir\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610043 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.609993    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-dshm\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610316 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.610069    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9ghs\" (UniqueName: \"kubernetes.io/projected/38add8ee-b4fb-4927-bca8-31103bceb5bc-kube-api-access-s9ghs\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610316 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.610153    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38add8ee-b4fb-4927-bca8-31103bceb5bc-tls-certs\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610316 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.610184    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-home\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.610332    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-model-cache\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610541 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.610447    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-tmp-dir\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610658 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.610584    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-home\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.610658 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.610631    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-model-cache\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.612500 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.612471    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-dshm\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.612753 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.612735    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38add8ee-b4fb-4927-bca8-31103bceb5bc-tls-certs\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.667077 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.667034    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9ghs\" (UniqueName: \"kubernetes.io/projected/38add8ee-b4fb-4927-bca8-31103bceb5bc-kube-api-access-s9ghs\") pod \"simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") " pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.678418 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.678372    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:55.756070 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.755613    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 18:58:55.756070 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:55.755675    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 18:58:56.061839 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:56.061813    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"]
Nov 02 18:58:56.063989 ip-10-0-129-72 kubenswrapper[4956]: W1102 18:58:56.063967    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38add8ee_b4fb_4927_bca8_31103bceb5bc.slice/crio-dc074afbaf130f48cd1a79842db423a46844a867cf8ee092cafb33792ae98558 WatchSource:0}: Error finding container dc074afbaf130f48cd1a79842db423a46844a867cf8ee092cafb33792ae98558: Status 404 returned error can't find the container with id dc074afbaf130f48cd1a79842db423a46844a867cf8ee092cafb33792ae98558
Nov 02 18:58:56.122245 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:56.122208    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9" event={"ID":"38add8ee-b4fb-4927-bca8-31103bceb5bc","Type":"ContainerStarted","Data":"dc074afbaf130f48cd1a79842db423a46844a867cf8ee092cafb33792ae98558"}
Nov 02 18:58:56.758349 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:56.758314    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:56.759250 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:56.759233    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 18:58:57.129715 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:57.129617    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9" event={"ID":"38add8ee-b4fb-4927-bca8-31103bceb5bc","Type":"ContainerStarted","Data":"319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257"}
Nov 02 18:58:57.243812 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:57.243760    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9" podStartSLOduration=2.243746324 podStartE2EDuration="2.243746324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:58:57.242051444 +0000 UTC m=+5767.499082062" watchObservedRunningTime="2026-11-02 18:58:57.243746324 +0000 UTC m=+5767.500776942"
Nov 02 18:58:57.679207 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:58:57.679168    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:58:59.052743 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:58:59.052703    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:59:05.679279 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:05.679240    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:59:05.691688 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:05.691664    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:59:06.188972 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:06.188938    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 18:59:29.053041 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:59:29.052999    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 18:59:56.501032 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:56.500996    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"]
Nov 02 18:59:57.297584 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:57.297547    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"]
Nov 02 18:59:57.405622 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:57.405581    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"]
Nov 02 18:59:59.008026 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:59.007987    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"]
Nov 02 18:59:59.008512 ip-10-0-129-72 kubenswrapper[4956]: I1102 18:59:59.008269    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4" podUID="1b47e4da-8d86-482c-95e0-69edb3143162" containerName="maas-api" containerID="cri-o://391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d" gracePeriod=30
Nov 02 18:59:59.052611 ip-10-0-129-72 kubenswrapper[4956]: E1102 18:59:59.052569    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:00:00.278230 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.278202    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 19:00:00.385673 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.385586    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b47e4da-8d86-482c-95e0-69edb3143162-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b47e4da-8d86-482c-95e0-69edb3143162-openshift-service-ca\") pod \"1b47e4da-8d86-482c-95e0-69edb3143162\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") "
Nov 02 19:00:00.385673 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.385646    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b47e4da-8d86-482c-95e0-69edb3143162-kube-api-access-xj8cn\" (UniqueName: \"kubernetes.io/projected/1b47e4da-8d86-482c-95e0-69edb3143162-kube-api-access-xj8cn\") pod \"1b47e4da-8d86-482c-95e0-69edb3143162\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") "
Nov 02 19:00:00.385673 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.385671    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-metrics-certs\") pod \"1b47e4da-8d86-482c-95e0-69edb3143162\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") "
Nov 02 19:00:00.385936 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.385721    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls\") pod \"1b47e4da-8d86-482c-95e0-69edb3143162\" (UID: \"1b47e4da-8d86-482c-95e0-69edb3143162\") "
Nov 02 19:00:00.386022 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.385999    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b47e4da-8d86-482c-95e0-69edb3143162-openshift-service-ca" pod "1b47e4da-8d86-482c-95e0-69edb3143162" (UID: "1b47e4da-8d86-482c-95e0-69edb3143162"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:00:00.388103 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.388067    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls" pod "1b47e4da-8d86-482c-95e0-69edb3143162" (UID: "1b47e4da-8d86-482c-95e0-69edb3143162"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:00:00.388103 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.388086    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-metrics-certs" pod "1b47e4da-8d86-482c-95e0-69edb3143162" (UID: "1b47e4da-8d86-482c-95e0-69edb3143162"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:00:00.388297 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.388141    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b47e4da-8d86-482c-95e0-69edb3143162-kube-api-access-xj8cn" pod "1b47e4da-8d86-482c-95e0-69edb3143162" (UID: "1b47e4da-8d86-482c-95e0-69edb3143162"). InnerVolumeSpecName "kube-api-access-xj8cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:00:00.459602 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.459571    4956 generic.go:356] "Generic (PLEG): container finished" podID="1b47e4da-8d86-482c-95e0-69edb3143162" containerID="391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d" exitCode=0
Nov 02 19:00:00.459791 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.459613    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4" event={"ID":"1b47e4da-8d86-482c-95e0-69edb3143162","Type":"ContainerDied","Data":"391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d"}
Nov 02 19:00:00.459791 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.459619    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"
Nov 02 19:00:00.459791 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.459657    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4" event={"ID":"1b47e4da-8d86-482c-95e0-69edb3143162","Type":"ContainerDied","Data":"2ba5fd7a0960016fe339ace9a677a2615134ad8ec68f1fb4e6a742f1e30b77b9"}
Nov 02 19:00:00.459791 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.459681    4956 scope.go:122] "RemoveContainer" containerID="391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d"
Nov 02 19:00:00.469896 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.469873    4956 scope.go:122] "RemoveContainer" containerID="391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d"
Nov 02 19:00:00.470250 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:00:00.470231    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d\": container with ID starting with 391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d not found: ID does not exist" containerID="391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d"
Nov 02 19:00:00.470306 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.470260    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d"} err="failed to get container status \"391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d\": rpc error: code = NotFound desc = could not find container \"391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d\": container with ID starting with 391e79f3a6465b74d334cbed045b7879c0604bbfd3ec7271d3c3e33d4656919d not found: ID does not exist"
Nov 02 19:00:00.487551 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.487515    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b47e4da-8d86-482c-95e0-69edb3143162-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:00.487551 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.487548    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj8cn\" (UniqueName: \"kubernetes.io/projected/1b47e4da-8d86-482c-95e0-69edb3143162-kube-api-access-xj8cn\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:00.487733 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.487560    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:00.487733 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.487569    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b47e4da-8d86-482c-95e0-69edb3143162-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:00.622114 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.622078    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"]
Nov 02 19:00:00.625783 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:00.625756    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6edebe-6c495868cd-rqxq4"]
Nov 02 19:00:01.863922 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:01.863887    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b47e4da-8d86-482c-95e0-69edb3143162" path="/var/lib/kubelet/pods/1b47e4da-8d86-482c-95e0-69edb3143162/volumes"
Nov 02 19:00:02.298798 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:02.298754    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" podUID="b9ce716a-b3c5-4655-8712-b5b593da3da0" containerName="payload-processing" containerID="cri-o://954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a" gracePeriod=25
Nov 02 19:00:02.406874 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:02.406803    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" podUID="bfddb85b-cecf-41b0-a648-b34a650ff227" containerName="payload-pre-processing" containerID="cri-o://0f31f73e2b87bb79a0a47b7935df3d76c322ba9dcb2f9176f4ab096ad8fbc912" gracePeriod=25
Nov 02 19:00:02.971193 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:02.971157    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" podUID="bfddb85b-cecf-41b0-a648-b34a650ff227" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:00:02.971607 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:02.971157    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" podUID="b9ce716a-b3c5-4655-8712-b5b593da3da0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:00:11.521492 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.521428    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9" podUID="38add8ee-b4fb-4927-bca8-31103bceb5bc" containerName="main" containerID="cri-o://319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257" gracePeriod=45
Nov 02 19:00:11.776247 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.776172    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 19:00:11.906656 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.906624    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-home\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-home\") pod \"38add8ee-b4fb-4927-bca8-31103bceb5bc\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") "
Nov 02 19:00:11.906861 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.906715    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38add8ee-b4fb-4927-bca8-31103bceb5bc-kube-api-access-s9ghs\" (UniqueName: \"kubernetes.io/projected/38add8ee-b4fb-4927-bca8-31103bceb5bc-kube-api-access-s9ghs\") pod \"38add8ee-b4fb-4927-bca8-31103bceb5bc\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") "
Nov 02 19:00:11.906861 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.906777    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-tmp-dir\") pod \"38add8ee-b4fb-4927-bca8-31103bceb5bc\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") "
Nov 02 19:00:11.906861 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.906805    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38add8ee-b4fb-4927-bca8-31103bceb5bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/38add8ee-b4fb-4927-bca8-31103bceb5bc-tls-certs\") pod \"38add8ee-b4fb-4927-bca8-31103bceb5bc\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") "
Nov 02 19:00:11.906861 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.906830    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-dshm\") pod \"38add8ee-b4fb-4927-bca8-31103bceb5bc\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") "
Nov 02 19:00:11.907079 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.906860    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-model-cache\") pod \"38add8ee-b4fb-4927-bca8-31103bceb5bc\" (UID: \"38add8ee-b4fb-4927-bca8-31103bceb5bc\") "
Nov 02 19:00:11.907079 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.906944    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-home" pod "38add8ee-b4fb-4927-bca8-31103bceb5bc" (UID: "38add8ee-b4fb-4927-bca8-31103bceb5bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:11.907282 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.907258    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:11.907400 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.907335    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-tmp-dir" pod "38add8ee-b4fb-4927-bca8-31103bceb5bc" (UID: "38add8ee-b4fb-4927-bca8-31103bceb5bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:11.907510 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.907367    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-model-cache" pod "38add8ee-b4fb-4927-bca8-31103bceb5bc" (UID: "38add8ee-b4fb-4927-bca8-31103bceb5bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:11.909339 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.909304    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38add8ee-b4fb-4927-bca8-31103bceb5bc-kube-api-access-s9ghs" pod "38add8ee-b4fb-4927-bca8-31103bceb5bc" (UID: "38add8ee-b4fb-4927-bca8-31103bceb5bc"). InnerVolumeSpecName "kube-api-access-s9ghs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:00:11.909491 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.909359    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38add8ee-b4fb-4927-bca8-31103bceb5bc-tls-certs" pod "38add8ee-b4fb-4927-bca8-31103bceb5bc" (UID: "38add8ee-b4fb-4927-bca8-31103bceb5bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:00:11.909591 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:11.909575    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-dshm" pod "38add8ee-b4fb-4927-bca8-31103bceb5bc" (UID: "38add8ee-b4fb-4927-bca8-31103bceb5bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:12.008694 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.008656    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:12.008694 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.008687    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38add8ee-b4fb-4927-bca8-31103bceb5bc-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:12.008694 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.008696    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:12.008945 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.008705    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38add8ee-b4fb-4927-bca8-31103bceb5bc-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:12.008945 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.008717    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9ghs\" (UniqueName: \"kubernetes.io/projected/38add8ee-b4fb-4927-bca8-31103bceb5bc-kube-api-access-s9ghs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:12.519289 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.519256    4956 generic.go:356] "Generic (PLEG): container finished" podID="38add8ee-b4fb-4927-bca8-31103bceb5bc" containerID="319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257" exitCode=0
Nov 02 19:00:12.519555 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.519302    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"
Nov 02 19:00:12.519555 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.519345    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9" event={"ID":"38add8ee-b4fb-4927-bca8-31103bceb5bc","Type":"ContainerDied","Data":"319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257"}
Nov 02 19:00:12.519555 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.519397    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9" event={"ID":"38add8ee-b4fb-4927-bca8-31103bceb5bc","Type":"ContainerDied","Data":"dc074afbaf130f48cd1a79842db423a46844a867cf8ee092cafb33792ae98558"}
Nov 02 19:00:12.519555 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.519422    4956 scope.go:122] "RemoveContainer" containerID="319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257"
Nov 02 19:00:12.529344 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.529215    4956 scope.go:122] "RemoveContainer" containerID="319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257"
Nov 02 19:00:12.529617 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:00:12.529529    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257\": container with ID starting with 319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257 not found: ID does not exist" containerID="319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257"
Nov 02 19:00:12.529617 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.529558    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257"} err="failed to get container status \"319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257\": rpc error: code = NotFound desc = could not find container \"319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257\": container with ID starting with 319c229b82c5f50a54a383b3e5cf675bb9037725a36bd63783f1180c2704a257 not found: ID does not exist"
Nov 02 19:00:12.683604 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.683563    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"]
Nov 02 19:00:12.685422 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.685398    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-6edebe/simulator-single-main-6edebe-kserve-fb6f99c7-w7qv9"]
Nov 02 19:00:12.970637 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.970587    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" podUID="bfddb85b-cecf-41b0-a648-b34a650ff227" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:00:12.970867 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:12.970831    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" podUID="b9ce716a-b3c5-4655-8712-b5b593da3da0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:00:13.863288 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:13.863254    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38add8ee-b4fb-4927-bca8-31103bceb5bc" path="/var/lib/kubelet/pods/38add8ee-b4fb-4927-bca8-31103bceb5bc/volumes"
Nov 02 19:00:22.558371 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.558293    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 19:00:22.569167 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.569099    4956 generic.go:356] "Generic (PLEG): container finished" podID="bfddb85b-cecf-41b0-a648-b34a650ff227" containerID="0f31f73e2b87bb79a0a47b7935df3d76c322ba9dcb2f9176f4ab096ad8fbc912" exitCode=0
Nov 02 19:00:22.569323 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.569161    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" event={"ID":"bfddb85b-cecf-41b0-a648-b34a650ff227","Type":"ContainerDied","Data":"0f31f73e2b87bb79a0a47b7935df3d76c322ba9dcb2f9176f4ab096ad8fbc912"}
Nov 02 19:00:22.570479 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.570460    4956 generic.go:356] "Generic (PLEG): container finished" podID="b9ce716a-b3c5-4655-8712-b5b593da3da0" containerID="954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a" exitCode=0
Nov 02 19:00:22.570578 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.570514    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"
Nov 02 19:00:22.570578 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.570573    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" event={"ID":"b9ce716a-b3c5-4655-8712-b5b593da3da0","Type":"ContainerDied","Data":"954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a"}
Nov 02 19:00:22.570661 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.570613    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6" event={"ID":"b9ce716a-b3c5-4655-8712-b5b593da3da0","Type":"ContainerDied","Data":"4609bf5b606ae94e3b330f30fa8269eaa354fafa34dff127e40acb6c04c1dd05"}
Nov 02 19:00:22.570661 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.570640    4956 scope.go:122] "RemoveContainer" containerID="954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a"
Nov 02 19:00:22.579644 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.579626    4956 scope.go:122] "RemoveContainer" containerID="954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a"
Nov 02 19:00:22.579928 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:00:22.579909    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a\": container with ID starting with 954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a not found: ID does not exist" containerID="954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a"
Nov 02 19:00:22.579999 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.579940    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a"} err="failed to get container status \"954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a\": rpc error: code = NotFound desc = could not find container \"954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a\": container with ID starting with 954fda5679936c462e909c24078fef8fbf00159c72f7c4329bbadcbcc4a37e0a not found: ID does not exist"
Nov 02 19:00:22.604626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.604524    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ce716a-b3c5-4655-8712-b5b593da3da0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ce716a-b3c5-4655-8712-b5b593da3da0-plugins-config-volume\") pod \"b9ce716a-b3c5-4655-8712-b5b593da3da0\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") "
Nov 02 19:00:22.604809 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.604660    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ce716a-b3c5-4655-8712-b5b593da3da0-kube-api-access-fq4xz\" (UniqueName: \"kubernetes.io/projected/b9ce716a-b3c5-4655-8712-b5b593da3da0-kube-api-access-fq4xz\") pod \"b9ce716a-b3c5-4655-8712-b5b593da3da0\" (UID: \"b9ce716a-b3c5-4655-8712-b5b593da3da0\") "
Nov 02 19:00:22.604873 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.604848    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ce716a-b3c5-4655-8712-b5b593da3da0-plugins-config-volume" pod "b9ce716a-b3c5-4655-8712-b5b593da3da0" (UID: "b9ce716a-b3c5-4655-8712-b5b593da3da0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:00:22.605070 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.605045    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ce716a-b3c5-4655-8712-b5b593da3da0-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:22.607042 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.607014    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ce716a-b3c5-4655-8712-b5b593da3da0-kube-api-access-fq4xz" pod "b9ce716a-b3c5-4655-8712-b5b593da3da0" (UID: "b9ce716a-b3c5-4655-8712-b5b593da3da0"). InnerVolumeSpecName "kube-api-access-fq4xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:00:22.656907 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.656882    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 19:00:22.705398 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.705367    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfddb85b-cecf-41b0-a648-b34a650ff227-kube-api-access-w9f8b\" (UniqueName: \"kubernetes.io/projected/bfddb85b-cecf-41b0-a648-b34a650ff227-kube-api-access-w9f8b\") pod \"bfddb85b-cecf-41b0-a648-b34a650ff227\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") "
Nov 02 19:00:22.705592 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.705438    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfddb85b-cecf-41b0-a648-b34a650ff227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfddb85b-cecf-41b0-a648-b34a650ff227-plugins-config-volume\") pod \"bfddb85b-cecf-41b0-a648-b34a650ff227\" (UID: \"bfddb85b-cecf-41b0-a648-b34a650ff227\") "
Nov 02 19:00:22.705720 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.705707    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4xz\" (UniqueName: \"kubernetes.io/projected/b9ce716a-b3c5-4655-8712-b5b593da3da0-kube-api-access-fq4xz\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:22.705909 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.705878    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfddb85b-cecf-41b0-a648-b34a650ff227-plugins-config-volume" pod "bfddb85b-cecf-41b0-a648-b34a650ff227" (UID: "bfddb85b-cecf-41b0-a648-b34a650ff227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:00:22.707619 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.707594    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfddb85b-cecf-41b0-a648-b34a650ff227-kube-api-access-w9f8b" pod "bfddb85b-cecf-41b0-a648-b34a650ff227" (UID: "bfddb85b-cecf-41b0-a648-b34a650ff227"). InnerVolumeSpecName "kube-api-access-w9f8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:00:22.806504 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.806473    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9f8b\" (UniqueName: \"kubernetes.io/projected/bfddb85b-cecf-41b0-a648-b34a650ff227-kube-api-access-w9f8b\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:22.806504 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.806502    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfddb85b-cecf-41b0-a648-b34a650ff227-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:22.989909 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.989873    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"]
Nov 02 19:00:22.991790 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:22.991773    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6edebe-7849fd57cs8pq6"]
Nov 02 19:00:23.576936 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:23.576851    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"
Nov 02 19:00:23.576936 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:23.576871    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf" event={"ID":"bfddb85b-cecf-41b0-a648-b34a650ff227","Type":"ContainerDied","Data":"e3bba2332f589e81a5842760d709b03b91e658e6834be9a349e18136af388937"}
Nov 02 19:00:23.576936 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:23.576930    4956 scope.go:122] "RemoveContainer" containerID="0f31f73e2b87bb79a0a47b7935df3d76c322ba9dcb2f9176f4ab096ad8fbc912"
Nov 02 19:00:23.740046 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:23.740005    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"]
Nov 02 19:00:23.790251 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:23.790212    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6edebe-874c5rg5zf"]
Nov 02 19:00:23.864246 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:23.864168    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ce716a-b3c5-4655-8712-b5b593da3da0" path="/var/lib/kubelet/pods/b9ce716a-b3c5-4655-8712-b5b593da3da0/volumes"
Nov 02 19:00:23.864559 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:23.864546    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfddb85b-cecf-41b0-a648-b34a650ff227" path="/var/lib/kubelet/pods/bfddb85b-cecf-41b0-a648-b34a650ff227/volumes"
Nov 02 19:00:29.052420 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:00:29.052383    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:00:37.465735 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:37.465697    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"]
Nov 02 19:00:37.466333 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:37.465964    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy" containerID="cri-o://8b03b7de2bb4957f2c967e1fe0386cfe14cc0cc74f146fc895a10f24b79302b5" gracePeriod=30
Nov 02 19:00:41.761135 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:41.761079    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 19:00:41.761530 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:41.761183    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:00:42.672515 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.672477    4956 generic.go:356] "Generic (PLEG): container finished" podID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerID="8b03b7de2bb4957f2c967e1fe0386cfe14cc0cc74f146fc895a10f24b79302b5" exitCode=0
Nov 02 19:00:42.672725 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.672532    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" event={"ID":"ddeb41e6-55ea-4267-9af1-137293b006fb","Type":"ContainerDied","Data":"8b03b7de2bb4957f2c967e1fe0386cfe14cc0cc74f146fc895a10f24b79302b5"}
Nov 02 19:00:42.672725 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.672598    4956 scope.go:122] "RemoveContainer" containerID="c7db7bd4206d71ba8ea086d51c121f6f826a67d2c4ce16907330f0c84457042e"
Nov 02 19:00:42.729870 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.729810    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 19:00:42.801176 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801112    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-podinfo\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801200    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-socket\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801242    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-certs\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801271    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-credential-socket\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801311    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddeb41e6-55ea-4267-9af1-137293b006fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddeb41e6-55ea-4267-9af1-137293b006fb-istiod-ca-cert\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801381    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-kube-api-access-mwwtj\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-kube-api-access-mwwtj\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801491    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-envoy\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801559    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-certs" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801584    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-data\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.801626 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801604    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-credential-socket" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:42.802071 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801577    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-socket" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:42.802071 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801713    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-token\") pod \"ddeb41e6-55ea-4267-9af1-137293b006fb\" (UID: \"ddeb41e6-55ea-4267-9af1-137293b006fb\") "
Nov 02 19:00:42.802071 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801757    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeb41e6-55ea-4267-9af1-137293b006fb-istiod-ca-cert" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:00:42.802071 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.801795    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-data" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:42.802521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.802260    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.802521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.802280    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.802521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.802295    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.802521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.802309    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.802521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.802323    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddeb41e6-55ea-4267-9af1-137293b006fb-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.803594 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.803566    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-podinfo" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:00:42.803730 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.803711    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-kube-api-access-mwwtj" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "kube-api-access-mwwtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:00:42.803983 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.803954    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-envoy" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:00:42.804076 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.804055    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-token" pod "ddeb41e6-55ea-4267-9af1-137293b006fb" (UID: "ddeb41e6-55ea-4267-9af1-137293b006fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:00:42.903408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.903370    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwtj\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-kube-api-access-mwwtj\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.903408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.903400    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.903408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.903410    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:42.903408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:42.903418    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddeb41e6-55ea-4267-9af1-137293b006fb-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:00:43.678710 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:43.678679    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"
Nov 02 19:00:43.678922 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:43.678719    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh" event={"ID":"ddeb41e6-55ea-4267-9af1-137293b006fb","Type":"ContainerDied","Data":"b5447dce054c3da366f972d731d005e51b70efb87537be8aaf90ce14ca59c6e0"}
Nov 02 19:00:43.678922 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:43.678783    4956 scope.go:122] "RemoveContainer" containerID="8b03b7de2bb4957f2c967e1fe0386cfe14cc0cc74f146fc895a10f24b79302b5"
Nov 02 19:00:43.883599 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:43.883565    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"]
Nov 02 19:00:43.932895 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:43.932813    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6edebe-openshift-default-85f4589d6fwzbbh"]
Nov 02 19:00:45.869869 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:45.869836    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" path="/var/lib/kubelet/pods/ddeb41e6-55ea-4267-9af1-137293b006fb/volumes"
Nov 02 19:00:53.504876 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.504841    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"]
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505251    4956 cpu_manager.go:414] "containerMap: removing container" podUID="b9ce716a-b3c5-4655-8712-b5b593da3da0" containerName="payload-processing"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505262    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ce716a-b3c5-4655-8712-b5b593da3da0" containerName="payload-processing"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505275    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1b47e4da-8d86-482c-95e0-69edb3143162" containerName="maas-api"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505281    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b47e4da-8d86-482c-95e0-69edb3143162" containerName="maas-api"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505288    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505293    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505299    4956 cpu_manager.go:414] "containerMap: removing container" podUID="bfddb85b-cecf-41b0-a648-b34a650ff227" containerName="payload-pre-processing"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505304    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfddb85b-cecf-41b0-a648-b34a650ff227" containerName="payload-pre-processing"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505315    4956 cpu_manager.go:414] "containerMap: removing container" podUID="38add8ee-b4fb-4927-bca8-31103bceb5bc" containerName="main"
Nov 02 19:00:53.505322 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505320    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38add8ee-b4fb-4927-bca8-31103bceb5bc" containerName="main"
Nov 02 19:00:53.505641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505405    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ce716a-b3c5-4655-8712-b5b593da3da0" containerName="payload-processing"
Nov 02 19:00:53.505641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505413    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b47e4da-8d86-482c-95e0-69edb3143162" containerName="maas-api"
Nov 02 19:00:53.505641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505419    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy"
Nov 02 19:00:53.505641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505423    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfddb85b-cecf-41b0-a648-b34a650ff227" containerName="payload-pre-processing"
Nov 02 19:00:53.505641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505430    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="38add8ee-b4fb-4927-bca8-31103bceb5bc" containerName="main"
Nov 02 19:00:53.505641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.505437    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy"
Nov 02 19:00:53.509980 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.509959    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.564148 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.564082    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"]
Nov 02 19:00:53.604564 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.604531    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-home\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.604745 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.604583    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-model-cache\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.604745 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.604606    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-dshm\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.604745 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.604686    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkth7\" (UniqueName: \"kubernetes.io/projected/4c4b55cf-69bb-450d-a676-7b814972c92d-kube-api-access-kkth7\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.604858 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.604751    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b55cf-69bb-450d-a676-7b814972c92d-tls-certs\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.604858 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.604805    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-tmp-dir\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.611256 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.611221    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ecd09c67-kserve-self-signed-certs\""
Nov 02 19:00:53.705441 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705404    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-tmp-dir\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705468    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-home\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705521    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-model-cache\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705548    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-dshm\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705581    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkth7\" (UniqueName: \"kubernetes.io/projected/4c4b55cf-69bb-450d-a676-7b814972c92d-kube-api-access-kkth7\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705810 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705632    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b55cf-69bb-450d-a676-7b814972c92d-tls-certs\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705903 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705882    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-tmp-dir\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705955 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705921    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-home\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.705955 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.705936    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-model-cache\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.708092 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.708063    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-dshm\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.708249 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.708232    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b55cf-69bb-450d-a676-7b814972c92d-tls-certs\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.762069 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.762000    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkth7\" (UniqueName: \"kubernetes.io/projected/4c4b55cf-69bb-450d-a676-7b814972c92d-kube-api-access-kkth7\") pod \"e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") " pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:53.823280 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:53.823239    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:54.194005 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:54.193955    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"]
Nov 02 19:00:54.196506 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:00:54.196476    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c4b55cf_69bb_450d_a676_7b814972c92d.slice/crio-db4da5ce514a8913b8ad99d5f3189fcde109ab28b53c352ed3a61787c871311e WatchSource:0}: Error finding container db4da5ce514a8913b8ad99d5f3189fcde109ab28b53c352ed3a61787c871311e: Status 404 returned error can't find the container with id db4da5ce514a8913b8ad99d5f3189fcde109ab28b53c352ed3a61787c871311e
Nov 02 19:00:54.735387 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:54.735347    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z" event={"ID":"4c4b55cf-69bb-450d-a676-7b814972c92d","Type":"ContainerStarted","Data":"1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6"}
Nov 02 19:00:54.735828 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:54.735399    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z" event={"ID":"4c4b55cf-69bb-450d-a676-7b814972c92d","Type":"ContainerStarted","Data":"db4da5ce514a8913b8ad99d5f3189fcde109ab28b53c352ed3a61787c871311e"}
Nov 02 19:00:54.824086 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:54.824045    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:00:54.848032 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:00:54.847918    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z" podStartSLOduration=1.847901456 podStartE2EDuration="1.847901456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:00:54.847757234 +0000 UTC m=+5885.104787855" watchObservedRunningTime="2026-11-02 19:00:54.847901456 +0000 UTC m=+5885.104932076"
Nov 02 19:00:59.058542 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:00:59.058507    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:01:03.824192 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:03.824092    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:01:03.836780 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:03.836749    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:01:04.826726 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:04.826697    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:01:21.091934 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.091898    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"]
Nov 02 19:01:21.092432 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.092349    4956 cpu_manager.go:414] "containerMap: removing container" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy"
Nov 02 19:01:21.092432 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.092361    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddeb41e6-55ea-4267-9af1-137293b006fb" containerName="istio-proxy"
Nov 02 19:01:21.095485 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.095465    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.198058 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.198024    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ecd09c67-kserve-self-signed-certs\""
Nov 02 19:01:21.200164 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.200134    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"]
Nov 02 19:01:21.223867 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.223835    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddjc\" (UniqueName: \"kubernetes.io/projected/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-kube-api-access-zddjc\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.224040 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.223888    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-dshm\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.224040 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.223911    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-home\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.224152 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.224045    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-model-cache\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.224152 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.224084    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tmp-dir\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.224238 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.224176    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tls-certs\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325152 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325087    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tls-certs\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325393 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325206    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddjc\" (UniqueName: \"kubernetes.io/projected/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-kube-api-access-zddjc\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325393 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325272    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-dshm\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325393 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325312    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-home\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325393 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325389    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-model-cache\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325705 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325412    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tmp-dir\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325810 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325780    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-home\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.325874 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.325826    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tmp-dir\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.326399 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.326371    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-model-cache\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.327860 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.327837    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-dshm\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.328074 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.328054    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tls-certs\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.382139 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.382046    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddjc\" (UniqueName: \"kubernetes.io/projected/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-kube-api-access-zddjc\") pod \"e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") " pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.408004 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.407968    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:21.779169 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.779143    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"]
Nov 02 19:01:21.780951 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:01:21.780918    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbd6d0e3_fb3d_4a99_8c9e_b7121327152e.slice/crio-e28058ce5c4cc557baa8df11a282129c74440fa7956b729d94e67da83481ed77 WatchSource:0}: Error finding container e28058ce5c4cc557baa8df11a282129c74440fa7956b729d94e67da83481ed77: Status 404 returned error can't find the container with id e28058ce5c4cc557baa8df11a282129c74440fa7956b729d94e67da83481ed77
Nov 02 19:01:21.894004 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.893952    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h" event={"ID":"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e","Type":"ContainerStarted","Data":"5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0"}
Nov 02 19:01:21.894004 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:21.894004    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h" event={"ID":"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e","Type":"ContainerStarted","Data":"e28058ce5c4cc557baa8df11a282129c74440fa7956b729d94e67da83481ed77"}
Nov 02 19:01:22.007063 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:22.007011    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h" podStartSLOduration=1.0069919 podStartE2EDuration="1.0069919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:01:22.00579264 +0000 UTC m=+5912.262823259" watchObservedRunningTime="2026-11-02 19:01:22.0069919 +0000 UTC m=+5912.264022521"
Nov 02 19:01:22.408994 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:22.408961    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:29.120447 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:01:29.120402    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:01:31.408628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:31.408592    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:31.421572 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:31.421546    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:31.966900 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:01:31.966870    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:01:59.053506 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:01:59.053464    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:02:03.561184 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:03.561146    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"]
Nov 02 19:02:08.157893 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:08.157853    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"]
Nov 02 19:02:18.582268 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.582198    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z" podUID="4c4b55cf-69bb-450d-a676-7b814972c92d" containerName="main" containerID="cri-o://1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6" gracePeriod=45
Nov 02 19:02:18.829777 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.829751    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:02:18.919641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.919550    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-home\") pod \"4c4b55cf-69bb-450d-a676-7b814972c92d\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") "
Nov 02 19:02:18.919641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.919624    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4b55cf-69bb-450d-a676-7b814972c92d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b55cf-69bb-450d-a676-7b814972c92d-tls-certs\") pod \"4c4b55cf-69bb-450d-a676-7b814972c92d\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") "
Nov 02 19:02:18.919898 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.919728    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4b55cf-69bb-450d-a676-7b814972c92d-kube-api-access-kkth7\" (UniqueName: \"kubernetes.io/projected/4c4b55cf-69bb-450d-a676-7b814972c92d-kube-api-access-kkth7\") pod \"4c4b55cf-69bb-450d-a676-7b814972c92d\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") "
Nov 02 19:02:18.919898 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.919830    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-home" pod "4c4b55cf-69bb-450d-a676-7b814972c92d" (UID: "4c4b55cf-69bb-450d-a676-7b814972c92d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:18.919898 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.919871    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-dshm\") pod \"4c4b55cf-69bb-450d-a676-7b814972c92d\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") "
Nov 02 19:02:18.919898 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.919907    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-tmp-dir\") pod \"4c4b55cf-69bb-450d-a676-7b814972c92d\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") "
Nov 02 19:02:18.920177 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.919971    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-model-cache\") pod \"4c4b55cf-69bb-450d-a676-7b814972c92d\" (UID: \"4c4b55cf-69bb-450d-a676-7b814972c92d\") "
Nov 02 19:02:18.920237 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.920198    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-tmp-dir" pod "4c4b55cf-69bb-450d-a676-7b814972c92d" (UID: "4c4b55cf-69bb-450d-a676-7b814972c92d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:18.920372 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.920354    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-model-cache" pod "4c4b55cf-69bb-450d-a676-7b814972c92d" (UID: "4c4b55cf-69bb-450d-a676-7b814972c92d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:18.920434 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.920360    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:18.920434 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.920390    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:18.921976 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.921950    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4b55cf-69bb-450d-a676-7b814972c92d-kube-api-access-kkth7" pod "4c4b55cf-69bb-450d-a676-7b814972c92d" (UID: "4c4b55cf-69bb-450d-a676-7b814972c92d"). InnerVolumeSpecName "kube-api-access-kkth7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:02:18.922502 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.922480    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-dshm" pod "4c4b55cf-69bb-450d-a676-7b814972c92d" (UID: "4c4b55cf-69bb-450d-a676-7b814972c92d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:18.922502 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:18.922486    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4b55cf-69bb-450d-a676-7b814972c92d-tls-certs" pod "4c4b55cf-69bb-450d-a676-7b814972c92d" (UID: "4c4b55cf-69bb-450d-a676-7b814972c92d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:02:19.021211 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.021176    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkth7\" (UniqueName: \"kubernetes.io/projected/4c4b55cf-69bb-450d-a676-7b814972c92d-kube-api-access-kkth7\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:19.021211 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.021208    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:19.021431 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.021220    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b55cf-69bb-450d-a676-7b814972c92d-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:19.021431 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.021230    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b55cf-69bb-450d-a676-7b814972c92d-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:19.232427 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.232392    4956 generic.go:356] "Generic (PLEG): container finished" podID="4c4b55cf-69bb-450d-a676-7b814972c92d" containerID="1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6" exitCode=0
Nov 02 19:02:19.232630 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.232435    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"
Nov 02 19:02:19.232630 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.232475    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z" event={"ID":"4c4b55cf-69bb-450d-a676-7b814972c92d","Type":"ContainerDied","Data":"1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6"}
Nov 02 19:02:19.232630 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.232526    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z" event={"ID":"4c4b55cf-69bb-450d-a676-7b814972c92d","Type":"ContainerDied","Data":"db4da5ce514a8913b8ad99d5f3189fcde109ab28b53c352ed3a61787c871311e"}
Nov 02 19:02:19.232630 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.232548    4956 scope.go:122] "RemoveContainer" containerID="1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6"
Nov 02 19:02:19.242237 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.242222    4956 scope.go:122] "RemoveContainer" containerID="1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6"
Nov 02 19:02:19.242508 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:02:19.242489    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6\": container with ID starting with 1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6 not found: ID does not exist" containerID="1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6"
Nov 02 19:02:19.242559 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.242517    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6"} err="failed to get container status \"1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6\": rpc error: code = NotFound desc = could not find container \"1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6\": container with ID starting with 1c812084cc542780c5027ce66e3a82a24f8f837f769f6047bbd8e0dbc0732df6 not found: ID does not exist"
Nov 02 19:02:19.394027 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.393988    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"]
Nov 02 19:02:19.397500 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.397477    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z"]
Nov 02 19:02:19.816179 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.816108    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-ecd09c67-kserve-7566644865-gwh2z" podUID="4c4b55cf-69bb-450d-a676-7b814972c92d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:02:19.863564 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:19.863529    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4b55cf-69bb-450d-a676-7b814972c92d" path="/var/lib/kubelet/pods/4c4b55cf-69bb-450d-a676-7b814972c92d/volumes"
Nov 02 19:02:23.178882 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.178841    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h" podUID="fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" containerName="main" containerID="cri-o://5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0" gracePeriod=45
Nov 02 19:02:23.446249 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.446188    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:02:23.568153 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.568087    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-kube-api-access-zddjc\" (UniqueName: \"kubernetes.io/projected/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-kube-api-access-zddjc\") pod \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") "
Nov 02 19:02:23.568356 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.568166    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-dshm\") pod \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") "
Nov 02 19:02:23.568356 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.568206    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-home\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-home\") pod \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") "
Nov 02 19:02:23.568356 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.568270    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tmp-dir\") pod \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") "
Nov 02 19:02:23.568356 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.568344    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-model-cache\") pod \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") "
Nov 02 19:02:23.568666 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.568380    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tls-certs\") pod \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\" (UID: \"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e\") "
Nov 02 19:02:23.568781 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.568744    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tmp-dir" pod "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" (UID: "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:23.570285 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.569262    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-home" pod "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" (UID: "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:23.570285 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.569580    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-model-cache" pod "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" (UID: "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:23.571234 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.571145    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-dshm" pod "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" (UID: "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:02:23.571234 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.571190    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-kube-api-access-zddjc" pod "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" (UID: "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e"). InnerVolumeSpecName "kube-api-access-zddjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:02:23.571492 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.571470    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tls-certs" pod "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" (UID: "fbd6d0e3-fb3d-4a99-8c9e-b7121327152e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:02:23.669813 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.669772    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:23.669813 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.669805    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:23.669813 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.669814    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:23.670082 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.669824    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:23.670082 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.669835    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:23.670082 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:23.669846    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddjc\" (UniqueName: \"kubernetes.io/projected/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e-kube-api-access-zddjc\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:02:24.261874 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.261839    4956 generic.go:356] "Generic (PLEG): container finished" podID="fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" containerID="5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0" exitCode=0
Nov 02 19:02:24.262363 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.261888    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"
Nov 02 19:02:24.262363 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.261921    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h" event={"ID":"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e","Type":"ContainerDied","Data":"5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0"}
Nov 02 19:02:24.262363 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.261968    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h" event={"ID":"fbd6d0e3-fb3d-4a99-8c9e-b7121327152e","Type":"ContainerDied","Data":"e28058ce5c4cc557baa8df11a282129c74440fa7956b729d94e67da83481ed77"}
Nov 02 19:02:24.262363 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.261985    4956 scope.go:122] "RemoveContainer" containerID="5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0"
Nov 02 19:02:24.270654 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.270634    4956 scope.go:122] "RemoveContainer" containerID="5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0"
Nov 02 19:02:24.270931 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:02:24.270911    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0\": container with ID starting with 5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0 not found: ID does not exist" containerID="5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0"
Nov 02 19:02:24.270997 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.270941    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0"} err="failed to get container status \"5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0\": rpc error: code = NotFound desc = could not find container \"5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0\": container with ID starting with 5222ecd7c5bab60b57a2832d1b8c20783815ba2f26afe6643f59dade133e08d0 not found: ID does not exist"
Nov 02 19:02:24.425176 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.425141    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"]
Nov 02 19:02:24.431325 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:24.431297    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ecd09c67-kserve-5f7954968f-4s24h"]
Nov 02 19:02:25.863253 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:25.863213    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" path="/var/lib/kubelet/pods/fbd6d0e3-fb3d-4a99-8c9e-b7121327152e/volumes"
Nov 02 19:02:29.053178 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:02:29.053144    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:02:33.398630 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:33.398227    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:02:33.398630 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:33.398291    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:02:50.717763 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:50.717735    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:02:50.727477 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:50.727455    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:02:52.945949 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.945915    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-664df499bc-ndllm"]
Nov 02 19:02:52.946397 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.946366    4956 cpu_manager.go:414] "containerMap: removing container" podUID="4c4b55cf-69bb-450d-a676-7b814972c92d" containerName="main"
Nov 02 19:02:52.946397 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.946387    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4b55cf-69bb-450d-a676-7b814972c92d" containerName="main"
Nov 02 19:02:52.946483 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.946420    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" containerName="main"
Nov 02 19:02:52.946483 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.946428    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" containerName="main"
Nov 02 19:02:52.947094 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.947047    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd6d0e3-fb3d-4a99-8c9e-b7121327152e" containerName="main"
Nov 02 19:02:52.947301 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.947145    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4b55cf-69bb-450d-a676-7b814972c92d" containerName="main"
Nov 02 19:02:52.956097 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.953900    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:52.957572 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.957546    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5r9c\" (UniqueName: \"kubernetes.io/projected/7dd43864-60fb-4635-b4d8-44252fc6acbf-kube-api-access-n5r9c\") pod \"payload-pre-processing-664df499bc-ndllm\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") " pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:52.957675 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:52.957593    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd43864-60fb-4635-b4d8-44252fc6acbf-plugins-config-volume\") pod \"payload-pre-processing-664df499bc-ndllm\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") " pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:53.005422 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:53.005382    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-664df499bc-ndllm"]
Nov 02 19:02:53.058850 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:53.058816    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd43864-60fb-4635-b4d8-44252fc6acbf-plugins-config-volume\") pod \"payload-pre-processing-664df499bc-ndllm\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") " pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:53.059015 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:53.058929    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5r9c\" (UniqueName: \"kubernetes.io/projected/7dd43864-60fb-4635-b4d8-44252fc6acbf-kube-api-access-n5r9c\") pod \"payload-pre-processing-664df499bc-ndllm\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") " pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:53.059597 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:53.059572    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd43864-60fb-4635-b4d8-44252fc6acbf-plugins-config-volume\") pod \"payload-pre-processing-664df499bc-ndllm\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") " pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:53.117642 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:53.117613    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5r9c\" (UniqueName: \"kubernetes.io/projected/7dd43864-60fb-4635-b4d8-44252fc6acbf-kube-api-access-n5r9c\") pod \"payload-pre-processing-664df499bc-ndllm\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") " pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:53.268962 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:53.268928    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:53.641207 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:53.641180    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-664df499bc-ndllm"]
Nov 02 19:02:53.643312 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:02:53.643268    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd43864_60fb_4635_b4d8_44252fc6acbf.slice/crio-ba990c47ce29a1eecf6990789ac172550c439bed7de81d2440bba7e87e2277a5 WatchSource:0}: Error finding container ba990c47ce29a1eecf6990789ac172550c439bed7de81d2440bba7e87e2277a5: Status 404 returned error can't find the container with id ba990c47ce29a1eecf6990789ac172550c439bed7de81d2440bba7e87e2277a5
Nov 02 19:02:54.437954 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:54.437915    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm" event={"ID":"7dd43864-60fb-4635-b4d8-44252fc6acbf","Type":"ContainerStarted","Data":"998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334"}
Nov 02 19:02:54.438507 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:54.437979    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:02:54.438507 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:54.438000    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm" event={"ID":"7dd43864-60fb-4635-b4d8-44252fc6acbf","Type":"ContainerStarted","Data":"ba990c47ce29a1eecf6990789ac172550c439bed7de81d2440bba7e87e2277a5"}
Nov 02 19:02:54.551962 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:02:54.551895    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm" podStartSLOduration=2.551875456 podStartE2EDuration="2.551875456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:02:54.551040284 +0000 UTC m=+6004.808070904" watchObservedRunningTime="2026-11-02 19:02:54.551875456 +0000 UTC m=+6004.808906077"
Nov 02 19:02:59.052022 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:02:59.051982    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:03:05.446003 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:05.445972    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:03:05.612519 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:05.612480    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"]
Nov 02 19:03:07.530130 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.530078    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-76fb74b98f-6c96m"]
Nov 02 19:03:07.533931 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.533913    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:07.589617 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.589586    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76fb74b98f-6c96m"]
Nov 02 19:03:07.600204 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.600174    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjs6z\" (UniqueName: \"kubernetes.io/projected/20a28ecf-c928-470d-8a89-27bb22d4b81d-kube-api-access-fjs6z\") pod \"payload-processing-76fb74b98f-6c96m\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") " pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:07.600384 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.600219    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a28ecf-c928-470d-8a89-27bb22d4b81d-plugins-config-volume\") pod \"payload-processing-76fb74b98f-6c96m\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") " pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:07.701149 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.701073    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjs6z\" (UniqueName: \"kubernetes.io/projected/20a28ecf-c928-470d-8a89-27bb22d4b81d-kube-api-access-fjs6z\") pod \"payload-processing-76fb74b98f-6c96m\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") " pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:07.701380 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.701185    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a28ecf-c928-470d-8a89-27bb22d4b81d-plugins-config-volume\") pod \"payload-processing-76fb74b98f-6c96m\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") " pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:07.701961 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.701938    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a28ecf-c928-470d-8a89-27bb22d4b81d-plugins-config-volume\") pod \"payload-processing-76fb74b98f-6c96m\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") " pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:07.758359 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.758329    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjs6z\" (UniqueName: \"kubernetes.io/projected/20a28ecf-c928-470d-8a89-27bb22d4b81d-kube-api-access-fjs6z\") pod \"payload-processing-76fb74b98f-6c96m\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") " pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:07.846773 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:07.846689    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:08.217009 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:08.216980    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76fb74b98f-6c96m"]
Nov 02 19:03:08.219713 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:03:08.219687    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a28ecf_c928_470d_8a89_27bb22d4b81d.slice/crio-ebd231853f339e14068431cb343da4ac61c47bf6fcca314cf23b04c27f1d6610 WatchSource:0}: Error finding container ebd231853f339e14068431cb343da4ac61c47bf6fcca314cf23b04c27f1d6610: Status 404 returned error can't find the container with id ebd231853f339e14068431cb343da4ac61c47bf6fcca314cf23b04c27f1d6610
Nov 02 19:03:08.508069 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:08.507971    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m" event={"ID":"20a28ecf-c928-470d-8a89-27bb22d4b81d","Type":"ContainerStarted","Data":"70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3"}
Nov 02 19:03:08.508069 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:08.508030    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m" event={"ID":"20a28ecf-c928-470d-8a89-27bb22d4b81d","Type":"ContainerStarted","Data":"ebd231853f339e14068431cb343da4ac61c47bf6fcca314cf23b04c27f1d6610"}
Nov 02 19:03:08.508069 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:08.508068    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:08.622270 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:08.622210    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m" podStartSLOduration=1.6221951030000001 podStartE2EDuration="1.622195103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:03:08.620602096 +0000 UTC m=+6018.877632716" watchObservedRunningTime="2026-11-02 19:03:08.622195103 +0000 UTC m=+6018.879225722"
Nov 02 19:03:10.613529 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:10.613483    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm" podUID="85b494d4-b1e9-4c28-941a-0eaee5637be8" containerName="payload-pre-processing" containerID="cri-o://3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78" gracePeriod=25
Nov 02 19:03:10.869435 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:10.869369    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 19:03:10.933624 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:10.933590    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b494d4-b1e9-4c28-941a-0eaee5637be8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b494d4-b1e9-4c28-941a-0eaee5637be8-plugins-config-volume\") pod \"85b494d4-b1e9-4c28-941a-0eaee5637be8\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") "
Nov 02 19:03:10.933805 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:10.933638    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b494d4-b1e9-4c28-941a-0eaee5637be8-kube-api-access-srvjb\" (UniqueName: \"kubernetes.io/projected/85b494d4-b1e9-4c28-941a-0eaee5637be8-kube-api-access-srvjb\") pod \"85b494d4-b1e9-4c28-941a-0eaee5637be8\" (UID: \"85b494d4-b1e9-4c28-941a-0eaee5637be8\") "
Nov 02 19:03:10.934071 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:10.934046    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b494d4-b1e9-4c28-941a-0eaee5637be8-plugins-config-volume" pod "85b494d4-b1e9-4c28-941a-0eaee5637be8" (UID: "85b494d4-b1e9-4c28-941a-0eaee5637be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:03:10.936012 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:10.935992    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b494d4-b1e9-4c28-941a-0eaee5637be8-kube-api-access-srvjb" pod "85b494d4-b1e9-4c28-941a-0eaee5637be8" (UID: "85b494d4-b1e9-4c28-941a-0eaee5637be8"). InnerVolumeSpecName "kube-api-access-srvjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:03:11.034960 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.034916    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b494d4-b1e9-4c28-941a-0eaee5637be8-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:11.034960 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.034953    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srvjb\" (UniqueName: \"kubernetes.io/projected/85b494d4-b1e9-4c28-941a-0eaee5637be8-kube-api-access-srvjb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:11.523028 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.522995    4956 generic.go:356] "Generic (PLEG): container finished" podID="85b494d4-b1e9-4c28-941a-0eaee5637be8" containerID="3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78" exitCode=0
Nov 02 19:03:11.523248 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.523037    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"
Nov 02 19:03:11.523248 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.523057    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm" event={"ID":"85b494d4-b1e9-4c28-941a-0eaee5637be8","Type":"ContainerDied","Data":"3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78"}
Nov 02 19:03:11.523248 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.523147    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm" event={"ID":"85b494d4-b1e9-4c28-941a-0eaee5637be8","Type":"ContainerDied","Data":"12672a586f19da99a194cda6f7d847f85caaaa4b2ace8dd5071cb929c956a264"}
Nov 02 19:03:11.523248 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.523169    4956 scope.go:122] "RemoveContainer" containerID="3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78"
Nov 02 19:03:11.532551 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.532531    4956 scope.go:122] "RemoveContainer" containerID="3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78"
Nov 02 19:03:11.532825 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:03:11.532806    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78\": container with ID starting with 3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78 not found: ID does not exist" containerID="3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78"
Nov 02 19:03:11.532879 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.532836    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78"} err="failed to get container status \"3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78\": rpc error: code = NotFound desc = could not find container \"3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78\": container with ID starting with 3eacb231740bd69ef1e6783fa4d658a8f8d1a54c1b2538073308601761d89d78 not found: ID does not exist"
Nov 02 19:03:11.684653 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.684609    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"]
Nov 02 19:03:11.733583 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.733542    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rs7wm"]
Nov 02 19:03:11.863262 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:11.863182    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b494d4-b1e9-4c28-941a-0eaee5637be8" path="/var/lib/kubelet/pods/85b494d4-b1e9-4c28-941a-0eaee5637be8/volumes"
Nov 02 19:03:19.516437 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:19.516399    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:19.656309 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:19.656272    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"]
Nov 02 19:03:21.763983 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.763946    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-587dbc7f89-kkwdk"]
Nov 02 19:03:21.764412 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.764390    4956 cpu_manager.go:414] "containerMap: removing container" podUID="85b494d4-b1e9-4c28-941a-0eaee5637be8" containerName="payload-pre-processing"
Nov 02 19:03:21.764412 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.764402    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b494d4-b1e9-4c28-941a-0eaee5637be8" containerName="payload-pre-processing"
Nov 02 19:03:21.764506 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.764466    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b494d4-b1e9-4c28-941a-0eaee5637be8" containerName="payload-pre-processing"
Nov 02 19:03:21.768740 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.768719    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:21.824228 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.824192    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-587dbc7f89-kkwdk"]
Nov 02 19:03:21.945480 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.945446    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50fa52e-8904-4af5-921f-0de0f05e63db-plugins-config-volume\") pod \"payload-processing-587dbc7f89-kkwdk\" (UID: \"f50fa52e-8904-4af5-921f-0de0f05e63db\") " pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:21.945665 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:21.945555    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj6qm\" (UniqueName: \"kubernetes.io/projected/f50fa52e-8904-4af5-921f-0de0f05e63db-kube-api-access-zj6qm\") pod \"payload-processing-587dbc7f89-kkwdk\" (UID: \"f50fa52e-8904-4af5-921f-0de0f05e63db\") " pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:22.046977 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:22.046880    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50fa52e-8904-4af5-921f-0de0f05e63db-plugins-config-volume\") pod \"payload-processing-587dbc7f89-kkwdk\" (UID: \"f50fa52e-8904-4af5-921f-0de0f05e63db\") " pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:22.046977 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:22.046944    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6qm\" (UniqueName: \"kubernetes.io/projected/f50fa52e-8904-4af5-921f-0de0f05e63db-kube-api-access-zj6qm\") pod \"payload-processing-587dbc7f89-kkwdk\" (UID: \"f50fa52e-8904-4af5-921f-0de0f05e63db\") " pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:22.047625 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:22.047601    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50fa52e-8904-4af5-921f-0de0f05e63db-plugins-config-volume\") pod \"payload-processing-587dbc7f89-kkwdk\" (UID: \"f50fa52e-8904-4af5-921f-0de0f05e63db\") " pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:22.103394 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:22.103364    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6qm\" (UniqueName: \"kubernetes.io/projected/f50fa52e-8904-4af5-921f-0de0f05e63db-kube-api-access-zj6qm\") pod \"payload-processing-587dbc7f89-kkwdk\" (UID: \"f50fa52e-8904-4af5-921f-0de0f05e63db\") " pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:22.382428 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:22.382334    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:22.756074 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:22.755770    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-587dbc7f89-kkwdk"]
Nov 02 19:03:22.758338 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:03:22.758312    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50fa52e_8904_4af5_921f_0de0f05e63db.slice/crio-375a3efb077d269fb680948307941b88300564a366e1a60c5a5750dbcd55d5f8 WatchSource:0}: Error finding container 375a3efb077d269fb680948307941b88300564a366e1a60c5a5750dbcd55d5f8: Status 404 returned error can't find the container with id 375a3efb077d269fb680948307941b88300564a366e1a60c5a5750dbcd55d5f8
Nov 02 19:03:23.583410 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:23.583370    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk" event={"ID":"f50fa52e-8904-4af5-921f-0de0f05e63db","Type":"ContainerStarted","Data":"c6011eea19c60855a6536effaa39310feb4aad3ab70fa8121d00f646822839e6"}
Nov 02 19:03:23.583828 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:23.583441    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:23.583828 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:23.583467    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk" event={"ID":"f50fa52e-8904-4af5-921f-0de0f05e63db","Type":"ContainerStarted","Data":"375a3efb077d269fb680948307941b88300564a366e1a60c5a5750dbcd55d5f8"}
Nov 02 19:03:23.697150 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:23.697081    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk" podStartSLOduration=2.69706825 podStartE2EDuration="2.69706825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:03:23.695384691 +0000 UTC m=+6033.952415305" watchObservedRunningTime="2026-11-02 19:03:23.69706825 +0000 UTC m=+6033.954098861"
Nov 02 19:03:24.656920 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:24.656874    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl" podUID="8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" containerName="payload-processing" containerID="cri-o://39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9" gracePeriod=25
Nov 02 19:03:24.909977 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:24.909912    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 19:03:25.074308 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.074271    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-plugins-config-volume\") pod \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") "
Nov 02 19:03:25.074472 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.074413    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-kube-api-access-mw88f\" (UniqueName: \"kubernetes.io/projected/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-kube-api-access-mw88f\") pod \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\" (UID: \"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44\") "
Nov 02 19:03:25.074755 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.074724    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-plugins-config-volume" pod "8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" (UID: "8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:03:25.076658 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.076639    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-kube-api-access-mw88f" pod "8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" (UID: "8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44"). InnerVolumeSpecName "kube-api-access-mw88f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:03:25.175188 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.175092    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw88f\" (UniqueName: \"kubernetes.io/projected/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-kube-api-access-mw88f\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:25.175188 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.175135    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:25.595102 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.595068    4956 generic.go:356] "Generic (PLEG): container finished" podID="8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" containerID="39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9" exitCode=0
Nov 02 19:03:25.595317 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.595115    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"
Nov 02 19:03:25.595317 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.595162    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl" event={"ID":"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44","Type":"ContainerDied","Data":"39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9"}
Nov 02 19:03:25.595317 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.595223    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gtcjl" event={"ID":"8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44","Type":"ContainerDied","Data":"93ec6a7a223fa88e89224f8750009b290e4eccecf4199a2959ced232157f3472"}
Nov 02 19:03:25.595317 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.595248    4956 scope.go:122] "RemoveContainer" containerID="39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9"
Nov 02 19:03:25.604225 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.604206    4956 scope.go:122] "RemoveContainer" containerID="39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9"
Nov 02 19:03:25.604494 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:03:25.604475    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9\": container with ID starting with 39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9 not found: ID does not exist" containerID="39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9"
Nov 02 19:03:25.604562 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.604502    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9"} err="failed to get container status \"39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9\": rpc error: code = NotFound desc = could not find container \"39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9\": container with ID starting with 39c1e526323033cb5aaa89f1a281615c6e1ecda3b46f73bf8cdaa114c28d0ac9 not found: ID does not exist"
Nov 02 19:03:25.757606 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.757561    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"]
Nov 02 19:03:25.807271 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.807238    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gtcjl"]
Nov 02 19:03:25.864135 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:25.864049    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" path="/var/lib/kubelet/pods/8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44/volumes"
Nov 02 19:03:29.056534 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:03:29.056492    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:03:34.141277 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:34.141240    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-664df499bc-ndllm"]
Nov 02 19:03:34.592221 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:34.592190    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-587dbc7f89-kkwdk"
Nov 02 19:03:34.727421 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:34.727388    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76fb74b98f-6c96m"]
Nov 02 19:03:39.141882 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.141834    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm" podUID="7dd43864-60fb-4635-b4d8-44252fc6acbf" containerName="payload-pre-processing" containerID="cri-o://998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334" gracePeriod=25
Nov 02 19:03:39.398230 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.398164    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:03:39.513971 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.513935    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd43864-60fb-4635-b4d8-44252fc6acbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd43864-60fb-4635-b4d8-44252fc6acbf-plugins-config-volume\") pod \"7dd43864-60fb-4635-b4d8-44252fc6acbf\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") "
Nov 02 19:03:39.514287 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.514024    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd43864-60fb-4635-b4d8-44252fc6acbf-kube-api-access-n5r9c\" (UniqueName: \"kubernetes.io/projected/7dd43864-60fb-4635-b4d8-44252fc6acbf-kube-api-access-n5r9c\") pod \"7dd43864-60fb-4635-b4d8-44252fc6acbf\" (UID: \"7dd43864-60fb-4635-b4d8-44252fc6acbf\") "
Nov 02 19:03:39.514613 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.514589    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd43864-60fb-4635-b4d8-44252fc6acbf-plugins-config-volume" pod "7dd43864-60fb-4635-b4d8-44252fc6acbf" (UID: "7dd43864-60fb-4635-b4d8-44252fc6acbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:03:39.516908 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.516881    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd43864-60fb-4635-b4d8-44252fc6acbf-kube-api-access-n5r9c" pod "7dd43864-60fb-4635-b4d8-44252fc6acbf" (UID: "7dd43864-60fb-4635-b4d8-44252fc6acbf"). InnerVolumeSpecName "kube-api-access-n5r9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:03:39.616034 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.616000    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd43864-60fb-4635-b4d8-44252fc6acbf-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:39.616034 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.616029    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5r9c\" (UniqueName: \"kubernetes.io/projected/7dd43864-60fb-4635-b4d8-44252fc6acbf-kube-api-access-n5r9c\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:39.663805 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.663716    4956 generic.go:356] "Generic (PLEG): container finished" podID="7dd43864-60fb-4635-b4d8-44252fc6acbf" containerID="998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334" exitCode=0
Nov 02 19:03:39.663805 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.663766    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm"
Nov 02 19:03:39.664023 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.663805    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm" event={"ID":"7dd43864-60fb-4635-b4d8-44252fc6acbf","Type":"ContainerDied","Data":"998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334"}
Nov 02 19:03:39.664023 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.663858    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-664df499bc-ndllm" event={"ID":"7dd43864-60fb-4635-b4d8-44252fc6acbf","Type":"ContainerDied","Data":"ba990c47ce29a1eecf6990789ac172550c439bed7de81d2440bba7e87e2277a5"}
Nov 02 19:03:39.664023 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.663878    4956 scope.go:122] "RemoveContainer" containerID="998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334"
Nov 02 19:03:39.673425 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.673406    4956 scope.go:122] "RemoveContainer" containerID="998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334"
Nov 02 19:03:39.673712 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:03:39.673695    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334\": container with ID starting with 998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334 not found: ID does not exist" containerID="998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334"
Nov 02 19:03:39.673763 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.673721    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334"} err="failed to get container status \"998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334\": rpc error: code = NotFound desc = could not find container \"998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334\": container with ID starting with 998eea3eef91ac2c0f3c34a5b1d9dd6585699cdba6fedd897c2d4a0ff8495334 not found: ID does not exist"
Nov 02 19:03:39.728054 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.727977    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m" podUID="20a28ecf-c928-470d-8a89-27bb22d4b81d" containerName="payload-processing" containerID="cri-o://70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3" gracePeriod=25
Nov 02 19:03:39.826571 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.826539    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-664df499bc-ndllm"]
Nov 02 19:03:39.833447 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.833416    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-664df499bc-ndllm"]
Nov 02 19:03:39.864316 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.864286    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd43864-60fb-4635-b4d8-44252fc6acbf" path="/var/lib/kubelet/pods/7dd43864-60fb-4635-b4d8-44252fc6acbf/volumes"
Nov 02 19:03:39.978080 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:39.978054    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:40.019264 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.019226    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a28ecf-c928-470d-8a89-27bb22d4b81d-kube-api-access-fjs6z\" (UniqueName: \"kubernetes.io/projected/20a28ecf-c928-470d-8a89-27bb22d4b81d-kube-api-access-fjs6z\") pod \"20a28ecf-c928-470d-8a89-27bb22d4b81d\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") "
Nov 02 19:03:40.019444 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.019289    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a28ecf-c928-470d-8a89-27bb22d4b81d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a28ecf-c928-470d-8a89-27bb22d4b81d-plugins-config-volume\") pod \"20a28ecf-c928-470d-8a89-27bb22d4b81d\" (UID: \"20a28ecf-c928-470d-8a89-27bb22d4b81d\") "
Nov 02 19:03:40.019639 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.019616    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a28ecf-c928-470d-8a89-27bb22d4b81d-plugins-config-volume" pod "20a28ecf-c928-470d-8a89-27bb22d4b81d" (UID: "20a28ecf-c928-470d-8a89-27bb22d4b81d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:03:40.021485 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.021464    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a28ecf-c928-470d-8a89-27bb22d4b81d-kube-api-access-fjs6z" pod "20a28ecf-c928-470d-8a89-27bb22d4b81d" (UID: "20a28ecf-c928-470d-8a89-27bb22d4b81d"). InnerVolumeSpecName "kube-api-access-fjs6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:03:40.120790 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.120751    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjs6z\" (UniqueName: \"kubernetes.io/projected/20a28ecf-c928-470d-8a89-27bb22d4b81d-kube-api-access-fjs6z\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:40.120790 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.120784    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a28ecf-c928-470d-8a89-27bb22d4b81d-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:03:40.670569 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.670529    4956 generic.go:356] "Generic (PLEG): container finished" podID="20a28ecf-c928-470d-8a89-27bb22d4b81d" containerID="70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3" exitCode=0
Nov 02 19:03:40.671084 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.670582    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m"
Nov 02 19:03:40.671084 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.670617    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m" event={"ID":"20a28ecf-c928-470d-8a89-27bb22d4b81d","Type":"ContainerDied","Data":"70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3"}
Nov 02 19:03:40.671084 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.670678    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76fb74b98f-6c96m" event={"ID":"20a28ecf-c928-470d-8a89-27bb22d4b81d","Type":"ContainerDied","Data":"ebd231853f339e14068431cb343da4ac61c47bf6fcca314cf23b04c27f1d6610"}
Nov 02 19:03:40.671084 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.670701    4956 scope.go:122] "RemoveContainer" containerID="70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3"
Nov 02 19:03:40.679674 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.679656    4956 scope.go:122] "RemoveContainer" containerID="70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3"
Nov 02 19:03:40.679948 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:03:40.679929    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3\": container with ID starting with 70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3 not found: ID does not exist" containerID="70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3"
Nov 02 19:03:40.680028 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.679956    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3"} err="failed to get container status \"70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3\": rpc error: code = NotFound desc = could not find container \"70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3\": container with ID starting with 70ccb4189f4e45c4ee304b2fccdaa2031bba46d59316e3c86e2f0620473498f3 not found: ID does not exist"
Nov 02 19:03:40.831993 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.831952    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76fb74b98f-6c96m"]
Nov 02 19:03:40.837705 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:40.837677    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-76fb74b98f-6c96m"]
Nov 02 19:03:41.106812 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.106772    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"]
Nov 02 19:03:41.107240 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107226    4956 cpu_manager.go:414] "containerMap: removing container" podUID="7dd43864-60fb-4635-b4d8-44252fc6acbf" containerName="payload-pre-processing"
Nov 02 19:03:41.107301 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107242    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd43864-60fb-4635-b4d8-44252fc6acbf" containerName="payload-pre-processing"
Nov 02 19:03:41.107301 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107269    4956 cpu_manager.go:414] "containerMap: removing container" podUID="8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" containerName="payload-processing"
Nov 02 19:03:41.107301 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107278    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" containerName="payload-processing"
Nov 02 19:03:41.107301 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107294    4956 cpu_manager.go:414] "containerMap: removing container" podUID="20a28ecf-c928-470d-8a89-27bb22d4b81d" containerName="payload-processing"
Nov 02 19:03:41.107301 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107299    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a28ecf-c928-470d-8a89-27bb22d4b81d" containerName="payload-processing"
Nov 02 19:03:41.107459 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107373    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="8750e627-3fc4-4ed2-9d3f-e5a4ff8f1c44" containerName="payload-processing"
Nov 02 19:03:41.107459 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107386    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a28ecf-c928-470d-8a89-27bb22d4b81d" containerName="payload-processing"
Nov 02 19:03:41.107459 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.107396    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd43864-60fb-4635-b4d8-44252fc6acbf" containerName="payload-pre-processing"
Nov 02 19:03:41.111752 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.111731    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.164633 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.164600    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 02 19:03:41.165185 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.165069    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xzt6h\""
Nov 02 19:03:41.167239 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.167214    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"]
Nov 02 19:03:41.213943 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.213898    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 19:03:41.214179 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.214047    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 19:03:41.214179 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.214048    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 19:03:41.214179 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.214094    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 19:03:41.231565 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.231534    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lx4\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-kube-api-access-k4lx4\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.231793 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.231599    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.231793 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.231728    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.231793 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.231759    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.333202 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.333164    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.333423 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.333212    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.333423 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.333325    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lx4\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-kube-api-access-k4lx4\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.333423 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.333405    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.334042 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.334017    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.334179 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.334090    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.335979 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.335957    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.391761 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.391685    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lx4\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-kube-api-access-k4lx4\") pod \"payload-processing-external-model-5bcd7997-6lzb7\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.424298 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.424257    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:41.796848 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.796819    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"]
Nov 02 19:03:41.799989 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:03:41.799960    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce8e128_75fb_4b4f_8468_29bf5ee6d20a.slice/crio-8f1b7e99be79158442452211491b16463ca1cf8ff6c22c40f7f09f3338fa676b WatchSource:0}: Error finding container 8f1b7e99be79158442452211491b16463ca1cf8ff6c22c40f7f09f3338fa676b: Status 404 returned error can't find the container with id 8f1b7e99be79158442452211491b16463ca1cf8ff6c22c40f7f09f3338fa676b
Nov 02 19:03:41.863759 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:41.863731    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a28ecf-c928-470d-8a89-27bb22d4b81d" path="/var/lib/kubelet/pods/20a28ecf-c928-470d-8a89-27bb22d4b81d/volumes"
Nov 02 19:03:42.685094 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:42.685038    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7" event={"ID":"fce8e128-75fb-4b4f-8468-29bf5ee6d20a","Type":"ContainerStarted","Data":"948e799bbc4d98404322f04a78d7965271ad528e82c6b7905807d80b51e2be08"}
Nov 02 19:03:42.685311 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:42.685153    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:42.685311 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:42.685182    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7" event={"ID":"fce8e128-75fb-4b4f-8468-29bf5ee6d20a","Type":"ContainerStarted","Data":"8f1b7e99be79158442452211491b16463ca1cf8ff6c22c40f7f09f3338fa676b"}
Nov 02 19:03:42.798936 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:42.798845    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7" podStartSLOduration=1.798823381 podStartE2EDuration="1.798823381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:03:42.798296731 +0000 UTC m=+6053.055327352" watchObservedRunningTime="2026-11-02 19:03:42.798823381 +0000 UTC m=+6053.055854002"
Nov 02 19:03:53.699740 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:03:53.699706    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:03:59.051975 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:03:59.051942    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:04:29.052280 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:04:29.052236    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:04:59.051827 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:04:59.051787    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:05:25.958716 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:25.958677    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"]
Nov 02 19:05:29.052505 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:05:29.052471    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:05:30.964207 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:30.963198    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7" podUID="fce8e128-75fb-4b4f-8468-29bf5ee6d20a" containerName="payload-processing" containerID="cri-o://948e799bbc4d98404322f04a78d7965271ad528e82c6b7905807d80b51e2be08" gracePeriod=25
Nov 02 19:05:31.210085 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.209955    4956 generic.go:356] "Generic (PLEG): container finished" podID="fce8e128-75fb-4b4f-8468-29bf5ee6d20a" containerID="948e799bbc4d98404322f04a78d7965271ad528e82c6b7905807d80b51e2be08" exitCode=0
Nov 02 19:05:31.210085 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.210034    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7" event={"ID":"fce8e128-75fb-4b4f-8468-29bf5ee6d20a","Type":"ContainerDied","Data":"948e799bbc4d98404322f04a78d7965271ad528e82c6b7905807d80b51e2be08"}
Nov 02 19:05:31.278931 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.278900    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:05:31.303481 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.303451    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-plugins-config-volume\") pod \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") "
Nov 02 19:05:31.303655 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.303533    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-routing-overlay-volume\") pod \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") "
Nov 02 19:05:31.303870 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.303846    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-plugins-config-volume" pod "fce8e128-75fb-4b4f-8468-29bf5ee6d20a" (UID: "fce8e128-75fb-4b4f-8468-29bf5ee6d20a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:05:31.303938 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.303902    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-routing-overlay-volume" pod "fce8e128-75fb-4b4f-8468-29bf5ee6d20a" (UID: "fce8e128-75fb-4b4f-8468-29bf5ee6d20a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:05:31.404784 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.404746    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-kube-api-access-k4lx4\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-kube-api-access-k4lx4\") pod \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") "
Nov 02 19:05:31.404983 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.404880    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-provider-credentials\") pod \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\" (UID: \"fce8e128-75fb-4b4f-8468-29bf5ee6d20a\") "
Nov 02 19:05:31.405057 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.405050    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:05:31.405153 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.405061    4956 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-routing-overlay-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:05:31.407093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.407064    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-provider-credentials" pod "fce8e128-75fb-4b4f-8468-29bf5ee6d20a" (UID: "fce8e128-75fb-4b4f-8468-29bf5ee6d20a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:05:31.407281 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.407209    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-kube-api-access-k4lx4" pod "fce8e128-75fb-4b4f-8468-29bf5ee6d20a" (UID: "fce8e128-75fb-4b4f-8468-29bf5ee6d20a"). InnerVolumeSpecName "kube-api-access-k4lx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:05:31.505904 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.505869    4956 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-provider-credentials\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:05:31.505904 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:31.505899    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4lx4\" (UniqueName: \"kubernetes.io/projected/fce8e128-75fb-4b4f-8468-29bf5ee6d20a-kube-api-access-k4lx4\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:05:32.215971 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:32.215932    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7" event={"ID":"fce8e128-75fb-4b4f-8468-29bf5ee6d20a","Type":"ContainerDied","Data":"8f1b7e99be79158442452211491b16463ca1cf8ff6c22c40f7f09f3338fa676b"}
Nov 02 19:05:32.216415 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:32.215981    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"
Nov 02 19:05:32.216415 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:32.216010    4956 scope.go:122] "RemoveContainer" containerID="948e799bbc4d98404322f04a78d7965271ad528e82c6b7905807d80b51e2be08"
Nov 02 19:05:32.377887 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:32.377850    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"]
Nov 02 19:05:32.382789 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:32.382760    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-6lzb7"]
Nov 02 19:05:33.863843 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:05:33.863803    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce8e128-75fb-4b4f-8468-29bf5ee6d20a" path="/var/lib/kubelet/pods/fce8e128-75fb-4b4f-8468-29bf5ee6d20a/volumes"
Nov 02 19:05:59.051860 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:05:59.051821    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:06:29.052184 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:06:29.052148    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:06:59.051656 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:06:59.051615    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:07:29.117821 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:07:29.117779    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:07:50.748248 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:07:50.748221    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:07:50.756819 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:07:50.756796    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:07:59.052764 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:07:59.052727    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:08:03.389294 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:08:03.389238    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:08:03.389996 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:08:03.389957    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:08:03.493280 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:08:03.493244    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:08:03.493494 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:08:03.493356    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:08:29.052278 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:08:29.052240    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:08:59.052243 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:08:59.052206    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:09:29.052387 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:09:29.052342    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:09:59.052177 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:09:59.052114    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:10:29.051942 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:10:29.051894    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:10:51.097149 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.094491    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"]
Nov 02 19:10:51.098522 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.098474    4956 cpu_manager.go:414] "containerMap: removing container" podUID="fce8e128-75fb-4b4f-8468-29bf5ee6d20a" containerName="payload-processing"
Nov 02 19:10:51.101009 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.100962    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce8e128-75fb-4b4f-8468-29bf5ee6d20a" containerName="payload-processing"
Nov 02 19:10:51.101210 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.101189    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce8e128-75fb-4b4f-8468-29bf5ee6d20a" containerName="payload-processing"
Nov 02 19:10:51.112245 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.110333    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.211044 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.211008    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-5b22de-openshift-default-dockercfg-zs494\""
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212632    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-credential-socket\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212692    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istiod-ca-cert\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212734    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-socket\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212782    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktg2\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-kube-api-access-kktg2\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212848    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-envoy\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212876    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-certs\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212916    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-token\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212953    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-podinfo\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.215093 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.212992    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-data\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.221150 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.221099    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"]
Nov 02 19:10:51.314402 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314348    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-credential-socket\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314421    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istiod-ca-cert\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314469    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-socket\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314518    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktg2\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-kube-api-access-kktg2\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314587    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-envoy\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314628 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314616    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-certs\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314953 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314659    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-token\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314953 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314697    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-podinfo\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.314953 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.314742    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-data\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.315157 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.315000    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-credential-socket\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.315232 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.315154    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-data\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.315896 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.315865    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-certs\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.316043 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.316016    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-socket\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.316290 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.316260    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istiod-ca-cert\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.318496 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.318445    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-podinfo\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.318767 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.318739    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-envoy\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.372641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.372550    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktg2\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-kube-api-access-kktg2\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.372641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.372565    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-token\") pod \"e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") " pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.438622 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.438585    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:51.864412 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.864356    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"]
Nov 02 19:10:51.865535 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:10:51.865508    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ceae3d1_a28d_42d7_abb5_87ce7a9031e4.slice/crio-47a9217ff2fff933345ee42bcefe584395fed0d069803f9b99e7213995e18ff5 WatchSource:0}: Error finding container 47a9217ff2fff933345ee42bcefe584395fed0d069803f9b99e7213995e18ff5: Status 404 returned error can't find the container with id 47a9217ff2fff933345ee42bcefe584395fed0d069803f9b99e7213995e18ff5
Nov 02 19:10:51.867468 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.867436    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 19:10:51.867576 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.867514    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 19:10:51.867576 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:51.867555    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 19:10:52.091841 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:52.091794    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" event={"ID":"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4","Type":"ContainerStarted","Data":"5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440"}
Nov 02 19:10:52.092046 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:52.091885    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" event={"ID":"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4","Type":"ContainerStarted","Data":"47a9217ff2fff933345ee42bcefe584395fed0d069803f9b99e7213995e18ff5"}
Nov 02 19:10:52.207312 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:52.207248    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" podStartSLOduration=1.207228096 podStartE2EDuration="1.207228096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:10:52.20556546 +0000 UTC m=+6482.462596081" watchObservedRunningTime="2026-11-02 19:10:52.207228096 +0000 UTC m=+6482.464258717"
Nov 02 19:10:52.439161 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:52.439105    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:52.444399 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:52.444371    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:53.096993 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:53.096963    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:53.098141 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:53.098106    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:10:58.709543 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.709503    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"]
Nov 02 19:10:58.714430 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.714412    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:58.766573 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.766543    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5b22de\""
Nov 02 19:10:58.768773 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.768740    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"]
Nov 02 19:10:58.792934 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.791815    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:58.793263 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.793237    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcsmh\" (UniqueName: \"kubernetes.io/projected/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-kube-api-access-xcsmh\") pod \"payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:58.815629 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.815597    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d751376d-dockercfg-fbq6g\""
Nov 02 19:10:58.861155 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.861105    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"]
Nov 02 19:10:58.864544 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.864529    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:58.894721 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.894682    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcsmh\" (UniqueName: \"kubernetes.io/projected/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-kube-api-access-xcsmh\") pod \"payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:58.894898 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.894809    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:58.895565 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.895539    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:58.919973 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.919936    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"]
Nov 02 19:10:58.951856 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.951822    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcsmh\" (UniqueName: \"kubernetes.io/projected/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-kube-api-access-xcsmh\") pod \"payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:58.995739 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.995664    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9rh\" (UniqueName: \"kubernetes.io/projected/e281c3c3-7c6f-4610-b4f2-82f28c865508-kube-api-access-mb9rh\") pod \"payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:58.995739 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:58.995731    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e281c3c3-7c6f-4610-b4f2-82f28c865508-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:59.028004 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.027971    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:10:59.058522 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:10:59.058489    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:10:59.097440 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.096922    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9rh\" (UniqueName: \"kubernetes.io/projected/e281c3c3-7c6f-4610-b4f2-82f28c865508-kube-api-access-mb9rh\") pod \"payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:59.097440 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.097004    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e281c3c3-7c6f-4610-b4f2-82f28c865508-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:59.098011 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.097986    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e281c3c3-7c6f-4610-b4f2-82f28c865508-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:59.153645 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.153613    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9rh\" (UniqueName: \"kubernetes.io/projected/e281c3c3-7c6f-4610-b4f2-82f28c865508-kube-api-access-mb9rh\") pod \"payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:59.179278 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.179251    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:10:59.397640 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.397615    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"]
Nov 02 19:10:59.399167 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:10:59.399140    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d996d2_d598_46d3_a1e9_fe2afe2d47b0.slice/crio-d660a7f5908a7345a749a22a7944e4e7a7e2c040b1518c597d8f3ba1990f5e7d WatchSource:0}: Error finding container d660a7f5908a7345a749a22a7944e4e7a7e2c040b1518c597d8f3ba1990f5e7d: Status 404 returned error can't find the container with id d660a7f5908a7345a749a22a7944e4e7a7e2c040b1518c597d8f3ba1990f5e7d
Nov 02 19:10:59.551157 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:10:59.551110    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"]
Nov 02 19:10:59.553180 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:10:59.553154    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode281c3c3_7c6f_4610_b4f2_82f28c865508.slice/crio-e625834001c93b7ec3f7a75678d9ab2e033495e30d6df57b4264956e0c5c356a WatchSource:0}: Error finding container e625834001c93b7ec3f7a75678d9ab2e033495e30d6df57b4264956e0c5c356a: Status 404 returned error can't find the container with id e625834001c93b7ec3f7a75678d9ab2e033495e30d6df57b4264956e0c5c356a
Nov 02 19:11:00.134602 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.134566    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" event={"ID":"e281c3c3-7c6f-4610-b4f2-82f28c865508","Type":"ContainerStarted","Data":"e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654"}
Nov 02 19:11:00.135300 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.134628    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" event={"ID":"e281c3c3-7c6f-4610-b4f2-82f28c865508","Type":"ContainerStarted","Data":"e625834001c93b7ec3f7a75678d9ab2e033495e30d6df57b4264956e0c5c356a"}
Nov 02 19:11:00.135300 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.134756    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:11:00.136082 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.136060    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" event={"ID":"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0","Type":"ContainerStarted","Data":"f638ebea532fe136ccdff10bed07e38d66ad77b302fd697f0043ca8d94fe6f51"}
Nov 02 19:11:00.136206 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.136103    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" event={"ID":"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0","Type":"ContainerStarted","Data":"d660a7f5908a7345a749a22a7944e4e7a7e2c040b1518c597d8f3ba1990f5e7d"}
Nov 02 19:11:00.136206 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.136159    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:11:00.248702 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.248650    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" podStartSLOduration=2.248636479 podStartE2EDuration="2.248636479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:11:00.24595904 +0000 UTC m=+6490.502989659" watchObservedRunningTime="2026-11-02 19:11:00.248636479 +0000 UTC m=+6490.505667098"
Nov 02 19:11:00.360873 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:00.360817    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" podStartSLOduration=2.360801697 podStartE2EDuration="2.360801697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:11:00.358311848 +0000 UTC m=+6490.615342468" watchObservedRunningTime="2026-11-02 19:11:00.360801697 +0000 UTC m=+6490.617832318"
Nov 02 19:11:11.152598 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:11.151102    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:11:11.152598 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:11.151353    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:11:18.226330 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:18.226295    4956 generic.go:356] "Generic (PLEG): container finished" podID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerID="5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440" exitCode=0
Nov 02 19:11:18.226869 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:18.226374    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" event={"ID":"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4","Type":"ContainerDied","Data":"5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440"}
Nov 02 19:11:18.226869 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:18.226814    4956 scope.go:122] "RemoveContainer" containerID="5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440"
Nov 02 19:11:18.228037 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:18.228008    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 19:11:18.228174 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:18.228153    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 19:11:18.228240 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:18.228215    4956 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 19:11:19.254275 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:19.254236    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" event={"ID":"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4","Type":"ContainerStarted","Data":"1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437"}
Nov 02 19:11:19.446425 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:19.446325    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:11:19.451196 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:19.451153    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 19:11:19.451366 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:19.451213    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 19:11:20.443618 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:20.443577    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 19:11:20.444278 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:20.444243    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 19:11:21.448754 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:21.448258    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 19:11:21.448754 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:21.448328    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 19:11:22.447149 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:22.446558    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 19:11:22.447149 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:22.446623    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 19:11:23.097889 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:23.097845    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:11:23.445208 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:23.442595    4956 patch_prober.go:29] interesting pod/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 19:11:23.445208 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:23.442661    4956 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 19:11:24.418577 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.418541    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"]
Nov 02 19:11:24.422263 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.422243    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.443107 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.443081    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:11:24.444040 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.444023    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:11:24.500868 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.500833    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-home\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.501087 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.501005    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-dshm\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.501087 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.501070    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tmp-dir\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.501273 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.501236    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tls-certs\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.501342 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.501285    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-model-cache\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.501388 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.501345    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tldx\" (UniqueName: \"kubernetes.io/projected/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-kube-api-access-7tldx\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.504964 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.504940    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5b22de\"/\"kube-root-ca.crt\""
Nov 02 19:11:24.525691 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.525665    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5b22de\"/\"openshift-service-ca.crt\""
Nov 02 19:11:24.525825 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.525774    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5b22de\"/\"simulator-single-main-5b22de-kserve-self-signed-certs\""
Nov 02 19:11:24.525825 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.525780    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5b22de\"/\"default-dockercfg-7t7bs\""
Nov 02 19:11:24.527687 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.527668    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"]
Nov 02 19:11:24.602192 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602111    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-dshm\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602387 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602215    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tmp-dir\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602387 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602265    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tls-certs\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602387 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602300    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-model-cache\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602387 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602331    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tldx\" (UniqueName: \"kubernetes.io/projected/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-kube-api-access-7tldx\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602577 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602397    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-home\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602647 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602621    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tmp-dir\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602715 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602679    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-model-cache\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.602868 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.602847    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-home\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.604610 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.604588    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-dshm\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.604923 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.604902    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tls-certs\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.658670 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.658640    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tldx\" (UniqueName: \"kubernetes.io/projected/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-kube-api-access-7tldx\") pod \"simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") " pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:24.733710 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:24.733661    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:25.113435 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:25.113392    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"]
Nov 02 19:11:25.115715 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:11:25.115682    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf70a7bf5_0431_46b4_aef9_94425fd0b9e7.slice/crio-53c549139b916eb979befebaa3ee03422ea2945635e3cc843463342cc0b5f21e WatchSource:0}: Error finding container 53c549139b916eb979befebaa3ee03422ea2945635e3cc843463342cc0b5f21e: Status 404 returned error can't find the container with id 53c549139b916eb979befebaa3ee03422ea2945635e3cc843463342cc0b5f21e
Nov 02 19:11:25.286726 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:25.286687    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns" event={"ID":"f70a7bf5-0431-46b4-aef9-94425fd0b9e7","Type":"ContainerStarted","Data":"9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57"}
Nov 02 19:11:25.286726 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:25.286731    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns" event={"ID":"f70a7bf5-0431-46b4-aef9-94425fd0b9e7","Type":"ContainerStarted","Data":"53c549139b916eb979befebaa3ee03422ea2945635e3cc843463342cc0b5f21e"}
Nov 02 19:11:25.402201 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:25.402079    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns" podStartSLOduration=1.402064534 podStartE2EDuration="1.402064534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:11:25.400000125 +0000 UTC m=+6515.657030756" watchObservedRunningTime="2026-11-02 19:11:25.402064534 +0000 UTC m=+6515.659095155"
Nov 02 19:11:25.734180 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:25.734142    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:29.052029 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:11:29.051992    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:11:34.734827 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:34.734738    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:34.748021 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:34.747989    4956 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:35.347714 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:11:35.347685    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:11:59.053213 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:11:59.053170    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:12:20.065164 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:20.065105    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"]
Nov 02 19:12:22.365295 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:22.365258    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"]
Nov 02 19:12:22.469662 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:22.469630    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"]
Nov 02 19:12:27.366215 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:27.366177    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" podUID="e281c3c3-7c6f-4610-b4f2-82f28c865508" containerName="payload-processing" containerID="cri-o://e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654" gracePeriod=25
Nov 02 19:12:27.470200 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:27.470153    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" podUID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" containerName="payload-pre-processing" containerID="cri-o://f638ebea532fe136ccdff10bed07e38d66ad77b302fd697f0043ca8d94fe6f51" gracePeriod=25
Nov 02 19:12:29.053012 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:12:29.052977    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:12:31.143408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:31.143364    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" podUID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:12:31.143824 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:31.143413    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" podUID="e281c3c3-7c6f-4610-b4f2-82f28c865508" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:12:35.085657 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.085586    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns" podUID="f70a7bf5-0431-46b4-aef9-94425fd0b9e7" containerName="main" containerID="cri-o://9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57" gracePeriod=45
Nov 02 19:12:35.344092 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.344030    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:12:35.400779 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.400749    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-home\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-home\") pod \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") "
Nov 02 19:12:35.400997 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.400794    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-model-cache\") pod \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") "
Nov 02 19:12:35.400997 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.400832    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tmp-dir\") pod \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") "
Nov 02 19:12:35.400997 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.400921    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tls-certs\") pod \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") "
Nov 02 19:12:35.400997 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.400975    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-dshm\") pod \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") "
Nov 02 19:12:35.401256 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.401003    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-kube-api-access-7tldx\" (UniqueName: \"kubernetes.io/projected/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-kube-api-access-7tldx\") pod \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\" (UID: \"f70a7bf5-0431-46b4-aef9-94425fd0b9e7\") "
Nov 02 19:12:35.401256 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.401061    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-home" pod "f70a7bf5-0431-46b4-aef9-94425fd0b9e7" (UID: "f70a7bf5-0431-46b4-aef9-94425fd0b9e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:12:35.401256 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.401070    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-model-cache" pod "f70a7bf5-0431-46b4-aef9-94425fd0b9e7" (UID: "f70a7bf5-0431-46b4-aef9-94425fd0b9e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:12:35.401256 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.401195    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tmp-dir" pod "f70a7bf5-0431-46b4-aef9-94425fd0b9e7" (UID: "f70a7bf5-0431-46b4-aef9-94425fd0b9e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:12:35.401457 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.401430    4956 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-home\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:35.401457 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.401445    4956 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-model-cache\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:35.401570 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.401471    4956 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tmp-dir\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:35.403418 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.403381    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-dshm" pod "f70a7bf5-0431-46b4-aef9-94425fd0b9e7" (UID: "f70a7bf5-0431-46b4-aef9-94425fd0b9e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:12:35.403418 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.403391    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tls-certs" pod "f70a7bf5-0431-46b4-aef9-94425fd0b9e7" (UID: "f70a7bf5-0431-46b4-aef9-94425fd0b9e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:12:35.403613 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.403525    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-kube-api-access-7tldx" pod "f70a7bf5-0431-46b4-aef9-94425fd0b9e7" (UID: "f70a7bf5-0431-46b4-aef9-94425fd0b9e7"). InnerVolumeSpecName "kube-api-access-7tldx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:12:35.502417 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.502380    4956 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-tls-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:35.502417 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.502415    4956 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-dshm\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:35.502620 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.502430    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tldx\" (UniqueName: \"kubernetes.io/projected/f70a7bf5-0431-46b4-aef9-94425fd0b9e7-kube-api-access-7tldx\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:35.625991 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.625905    4956 generic.go:356] "Generic (PLEG): container finished" podID="f70a7bf5-0431-46b4-aef9-94425fd0b9e7" containerID="9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57" exitCode=0
Nov 02 19:12:35.625991 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.625949    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns" event={"ID":"f70a7bf5-0431-46b4-aef9-94425fd0b9e7","Type":"ContainerDied","Data":"9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57"}
Nov 02 19:12:35.625991 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.625951    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"
Nov 02 19:12:35.625991 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.625978    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns" event={"ID":"f70a7bf5-0431-46b4-aef9-94425fd0b9e7","Type":"ContainerDied","Data":"53c549139b916eb979befebaa3ee03422ea2945635e3cc843463342cc0b5f21e"}
Nov 02 19:12:35.626335 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.625996    4956 scope.go:122] "RemoveContainer" containerID="9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57"
Nov 02 19:12:35.635243 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.635224    4956 scope.go:122] "RemoveContainer" containerID="9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57"
Nov 02 19:12:35.635518 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:12:35.635493    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57\": container with ID starting with 9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57 not found: ID does not exist" containerID="9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57"
Nov 02 19:12:35.635576 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.635530    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57"} err="failed to get container status \"9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57\": rpc error: code = NotFound desc = could not find container \"9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57\": container with ID starting with 9204b9cc5fe801ed26c35e55025f414294e162f68971fcec73cf3433fc46fe57 not found: ID does not exist"
Nov 02 19:12:35.786775 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.786740    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"]
Nov 02 19:12:35.792019 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.791989    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns"]
Nov 02 19:12:35.864098 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:35.864061    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70a7bf5-0431-46b4-aef9-94425fd0b9e7" path="/var/lib/kubelet/pods/f70a7bf5-0431-46b4-aef9-94425fd0b9e7/volumes"
Nov 02 19:12:36.338072 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:36.338025    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-5b22de/simulator-single-main-5b22de-kserve-fcc5cbffd-7r2ns" podUID="f70a7bf5-0431-46b4-aef9-94425fd0b9e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:12:41.142927 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:41.142881    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" podUID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:12:41.142927 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:41.142913    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" podUID="e281c3c3-7c6f-4610-b4f2-82f28c865508" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 19:12:47.614431 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.614403    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:12:47.686842 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.686809    4956 generic.go:356] "Generic (PLEG): container finished" podID="e281c3c3-7c6f-4610-b4f2-82f28c865508" containerID="e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654" exitCode=0
Nov 02 19:12:47.687041 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.686876    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"
Nov 02 19:12:47.687041 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.686875    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" event={"ID":"e281c3c3-7c6f-4610-b4f2-82f28c865508","Type":"ContainerDied","Data":"e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654"}
Nov 02 19:12:47.687041 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.686971    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q" event={"ID":"e281c3c3-7c6f-4610-b4f2-82f28c865508","Type":"ContainerDied","Data":"e625834001c93b7ec3f7a75678d9ab2e033495e30d6df57b4264956e0c5c356a"}
Nov 02 19:12:47.687041 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.687002    4956 scope.go:122] "RemoveContainer" containerID="e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654"
Nov 02 19:12:47.689638 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.689615    4956 generic.go:356] "Generic (PLEG): container finished" podID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" containerID="f638ebea532fe136ccdff10bed07e38d66ad77b302fd697f0043ca8d94fe6f51" exitCode=0
Nov 02 19:12:47.689768 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.689671    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" event={"ID":"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0","Type":"ContainerDied","Data":"f638ebea532fe136ccdff10bed07e38d66ad77b302fd697f0043ca8d94fe6f51"}
Nov 02 19:12:47.696407 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.696388    4956 scope.go:122] "RemoveContainer" containerID="e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654"
Nov 02 19:12:47.696689 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:12:47.696670    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654\": container with ID starting with e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654 not found: ID does not exist" containerID="e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654"
Nov 02 19:12:47.696769 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.696698    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654"} err="failed to get container status \"e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654\": rpc error: code = NotFound desc = could not find container \"e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654\": container with ID starting with e66028303926612bd22688ef4e5e97671436f88859673b2f0257904ccf98f654 not found: ID does not exist"
Nov 02 19:12:47.723063 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.723041    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e281c3c3-7c6f-4610-b4f2-82f28c865508-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e281c3c3-7c6f-4610-b4f2-82f28c865508-plugins-config-volume\") pod \"e281c3c3-7c6f-4610-b4f2-82f28c865508\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") "
Nov 02 19:12:47.723198 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.723182    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e281c3c3-7c6f-4610-b4f2-82f28c865508-kube-api-access-mb9rh\" (UniqueName: \"kubernetes.io/projected/e281c3c3-7c6f-4610-b4f2-82f28c865508-kube-api-access-mb9rh\") pod \"e281c3c3-7c6f-4610-b4f2-82f28c865508\" (UID: \"e281c3c3-7c6f-4610-b4f2-82f28c865508\") "
Nov 02 19:12:47.723408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.723383    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e281c3c3-7c6f-4610-b4f2-82f28c865508-plugins-config-volume" pod "e281c3c3-7c6f-4610-b4f2-82f28c865508" (UID: "e281c3c3-7c6f-4610-b4f2-82f28c865508"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:12:47.723515 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.723497    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e281c3c3-7c6f-4610-b4f2-82f28c865508-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:47.725328 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.725295    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e281c3c3-7c6f-4610-b4f2-82f28c865508-kube-api-access-mb9rh" pod "e281c3c3-7c6f-4610-b4f2-82f28c865508" (UID: "e281c3c3-7c6f-4610-b4f2-82f28c865508"). InnerVolumeSpecName "kube-api-access-mb9rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:12:47.748810 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.748777    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:12:47.823812 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.823786    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-kube-api-access-xcsmh\" (UniqueName: \"kubernetes.io/projected/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-kube-api-access-xcsmh\") pod \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") "
Nov 02 19:12:47.823986 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.823825    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-plugins-config-volume\") pod \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\" (UID: \"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0\") "
Nov 02 19:12:47.824097 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.824083    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9rh\" (UniqueName: \"kubernetes.io/projected/e281c3c3-7c6f-4610-b4f2-82f28c865508-kube-api-access-mb9rh\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:47.824239 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.824209    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-plugins-config-volume" pod "e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" (UID: "e0d996d2-d598-46d3-a1e9-fe2afe2d47b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:12:47.826038 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.826013    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-kube-api-access-xcsmh" pod "e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" (UID: "e0d996d2-d598-46d3-a1e9-fe2afe2d47b0"). InnerVolumeSpecName "kube-api-access-xcsmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:12:47.924699 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.924621    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcsmh\" (UniqueName: \"kubernetes.io/projected/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-kube-api-access-xcsmh\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:47.924699 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:47.924648    4956 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0-plugins-config-volume\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:48.100110 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:48.100074    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"]
Nov 02 19:12:48.102905 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:48.102881    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5b22de-78c57c8cfrgx4q"]
Nov 02 19:12:48.696689 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:48.696600    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92" event={"ID":"e0d996d2-d598-46d3-a1e9-fe2afe2d47b0","Type":"ContainerDied","Data":"d660a7f5908a7345a749a22a7944e4e7a7e2c040b1518c597d8f3ba1990f5e7d"}
Nov 02 19:12:48.696689 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:48.696624    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"
Nov 02 19:12:48.696689 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:48.696657    4956 scope.go:122] "RemoveContainer" containerID="f638ebea532fe136ccdff10bed07e38d66ad77b302fd697f0043ca8d94fe6f51"
Nov 02 19:12:48.858102 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:48.858064    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"]
Nov 02 19:12:48.907391 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:48.907361    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-5b22de-67f89qph92"]
Nov 02 19:12:49.863701 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:49.863673    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" path="/var/lib/kubelet/pods/e0d996d2-d598-46d3-a1e9-fe2afe2d47b0/volumes"
Nov 02 19:12:49.864090 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:49.864033    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e281c3c3-7c6f-4610-b4f2-82f28c865508" path="/var/lib/kubelet/pods/e281c3c3-7c6f-4610-b4f2-82f28c865508/volumes"
Nov 02 19:12:50.782361 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:50.782322    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:12:50.790472 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:12:50.790450    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:12:59.052585 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:12:59.052549    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:13:00.139894 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.139807    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"]
Nov 02 19:13:00.140308 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140249    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" containerName="payload-pre-processing"
Nov 02 19:13:00.140308 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140264    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" containerName="payload-pre-processing"
Nov 02 19:13:00.140308 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140275    4956 cpu_manager.go:414] "containerMap: removing container" podUID="e281c3c3-7c6f-4610-b4f2-82f28c865508" containerName="payload-processing"
Nov 02 19:13:00.140308 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140281    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e281c3c3-7c6f-4610-b4f2-82f28c865508" containerName="payload-processing"
Nov 02 19:13:00.140444 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140316    4956 cpu_manager.go:414] "containerMap: removing container" podUID="f70a7bf5-0431-46b4-aef9-94425fd0b9e7" containerName="main"
Nov 02 19:13:00.140444 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140324    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70a7bf5-0431-46b4-aef9-94425fd0b9e7" containerName="main"
Nov 02 19:13:00.140444 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140383    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e281c3c3-7c6f-4610-b4f2-82f28c865508" containerName="payload-processing"
Nov 02 19:13:00.140444 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140398    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70a7bf5-0431-46b4-aef9-94425fd0b9e7" containerName="main"
Nov 02 19:13:00.140444 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.140405    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d996d2-d598-46d3-a1e9-fe2afe2d47b0" containerName="payload-pre-processing"
Nov 02 19:13:00.145065 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.145046    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.196984 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.196952    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5b22de-serving-cert\""
Nov 02 19:13:00.248291 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.248254    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"]
Nov 02 19:13:00.248495 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.248310    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-kube-api-access-jx8wh\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.248495 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.248358    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.248495 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.248477    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.248645 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.248606    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.350159 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.350104    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.350374 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.350237    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-kube-api-access-jx8wh\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.350374 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.350272    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.350374 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.350320    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.350570 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:13:00.350472    4956 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-serving-cert: secret "maas-api-e2e-worker-single-main-5b22de-serving-cert" not found
Nov 02 19:13:00.350570 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:13:00.350565    4956 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls podName:a9aa805b-7fa8-456a-b7d6-ec9757a83fac nodeName:}" failed. No retries permitted until 2026-11-02 19:13:00.850541198 +0000 UTC m=+6611.107571819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls") pod "maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2" (UID: "a9aa805b-7fa8-456a-b7d6-ec9757a83fac") : secret "maas-api-e2e-worker-single-main-5b22de-serving-cert" not found
Nov 02 19:13:00.350798 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.350775    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.353168 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.353141    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.409094 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.409028    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-kube-api-access-jx8wh\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.855945 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.855907    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:00.858983 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:00.858947    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:01.057949 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:01.057911    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:01.429857 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:01.429783    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"]
Nov 02 19:13:01.432354 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:13:01.432327    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aa805b_7fa8_456a_b7d6_ec9757a83fac.slice/crio-804012c468d15ef6446b7c33eabf1892082ad52f7e614ab5e6035f90aaf907fd WatchSource:0}: Error finding container 804012c468d15ef6446b7c33eabf1892082ad52f7e614ab5e6035f90aaf907fd: Status 404 returned error can't find the container with id 804012c468d15ef6446b7c33eabf1892082ad52f7e614ab5e6035f90aaf907fd
Nov 02 19:13:01.433840 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:01.433818    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:13:01.771169 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:01.771133    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2" event={"ID":"a9aa805b-7fa8-456a-b7d6-ec9757a83fac","Type":"ContainerStarted","Data":"804012c468d15ef6446b7c33eabf1892082ad52f7e614ab5e6035f90aaf907fd"}
Nov 02 19:13:02.777299 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:02.777253    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2" event={"ID":"a9aa805b-7fa8-456a-b7d6-ec9757a83fac","Type":"ContainerStarted","Data":"025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b"}
Nov 02 19:13:02.777794 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:02.777327    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:02.890382 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:02.890317    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2" podStartSLOduration=2.009254142 podStartE2EDuration="2.890302422s" totalImagesPullingTime="881.04828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:13:02.889104512 +0000 UTC m=+6613.146135132" watchObservedRunningTime="2026-11-02 19:13:02.890302422 +0000 UTC m=+6613.147333042"
Nov 02 19:13:08.788747 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:08.788716    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:28.125570 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.125536    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"]
Nov 02 19:13:28.126177 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.125731    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2" podUID="a9aa805b-7fa8-456a-b7d6-ec9757a83fac" containerName="maas-api" containerID="cri-o://025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b" gracePeriod=30
Nov 02 19:13:28.397965 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.397941    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:28.526035 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.526005    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-openshift-service-ca\") pod \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") "
Nov 02 19:13:28.526267 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.526091    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls\") pod \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") "
Nov 02 19:13:28.526267 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.526168    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-kube-api-access-jx8wh\") pod \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") "
Nov 02 19:13:28.526267 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.526204    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-metrics-certs\") pod \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\" (UID: \"a9aa805b-7fa8-456a-b7d6-ec9757a83fac\") "
Nov 02 19:13:28.526478 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.526366    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-openshift-service-ca" pod "a9aa805b-7fa8-456a-b7d6-ec9757a83fac" (UID: "a9aa805b-7fa8-456a-b7d6-ec9757a83fac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:13:28.526598 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.526574    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:28.528461 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.528427    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-metrics-certs" pod "a9aa805b-7fa8-456a-b7d6-ec9757a83fac" (UID: "a9aa805b-7fa8-456a-b7d6-ec9757a83fac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:13:28.528578 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.528490    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls" pod "a9aa805b-7fa8-456a-b7d6-ec9757a83fac" (UID: "a9aa805b-7fa8-456a-b7d6-ec9757a83fac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:13:28.528578 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.528508    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-kube-api-access-jx8wh" pod "a9aa805b-7fa8-456a-b7d6-ec9757a83fac" (UID: "a9aa805b-7fa8-456a-b7d6-ec9757a83fac"). InnerVolumeSpecName "kube-api-access-jx8wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:13:28.628246 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.628205    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:28.628246 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.628245    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-kube-api-access-jx8wh\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:28.628630 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.628263    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9aa805b-7fa8-456a-b7d6-ec9757a83fac-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:28.898561 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.898521    4956 generic.go:356] "Generic (PLEG): container finished" podID="a9aa805b-7fa8-456a-b7d6-ec9757a83fac" containerID="025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b" exitCode=0
Nov 02 19:13:28.898786 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.898571    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"
Nov 02 19:13:28.898786 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.898604    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2" event={"ID":"a9aa805b-7fa8-456a-b7d6-ec9757a83fac","Type":"ContainerDied","Data":"025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b"}
Nov 02 19:13:28.898786 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.898678    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2" event={"ID":"a9aa805b-7fa8-456a-b7d6-ec9757a83fac","Type":"ContainerDied","Data":"804012c468d15ef6446b7c33eabf1892082ad52f7e614ab5e6035f90aaf907fd"}
Nov 02 19:13:28.898786 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.898709    4956 scope.go:122] "RemoveContainer" containerID="025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b"
Nov 02 19:13:28.911296 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.911243    4956 scope.go:122] "RemoveContainer" containerID="025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b"
Nov 02 19:13:28.911650 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:13:28.911622    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b\": container with ID starting with 025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b not found: ID does not exist" containerID="025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b"
Nov 02 19:13:28.911728 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:28.911663    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b"} err="failed to get container status \"025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b\": rpc error: code = NotFound desc = could not find container \"025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b\": container with ID starting with 025a0d844b22489a9591a926dd9e4689009733e56744db5d80be214e58b4cf8b not found: ID does not exist"
Nov 02 19:13:29.054925 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:13:29.054889    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:13:29.060051 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:29.060020    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"]
Nov 02 19:13:29.064365 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:29.064339    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-8hjr2"]
Nov 02 19:13:29.864083 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:29.864049    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9aa805b-7fa8-456a-b7d6-ec9757a83fac" path="/var/lib/kubelet/pods/a9aa805b-7fa8-456a-b7d6-ec9757a83fac/volumes"
Nov 02 19:13:30.608473 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:30.608434    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"]
Nov 02 19:13:30.608667 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:30.608635    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy" containerID="cri-o://1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437" gracePeriod=30
Nov 02 19:13:35.874659 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.874592    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:13:35.931895 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.931862    4956 generic.go:356] "Generic (PLEG): container finished" podID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerID="1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437" exitCode=0
Nov 02 19:13:35.932083 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.931900    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" event={"ID":"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4","Type":"ContainerDied","Data":"1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437"}
Nov 02 19:13:35.932083 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.931919    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"
Nov 02 19:13:35.932083 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.931942    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx" event={"ID":"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4","Type":"ContainerDied","Data":"47a9217ff2fff933345ee42bcefe584395fed0d069803f9b99e7213995e18ff5"}
Nov 02 19:13:35.932083 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.931964    4956 scope.go:122] "RemoveContainer" containerID="1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437"
Nov 02 19:13:35.940744 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.940723    4956 scope.go:122] "RemoveContainer" containerID="5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440"
Nov 02 19:13:35.948939 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.948922    4956 scope.go:122] "RemoveContainer" containerID="1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437"
Nov 02 19:13:35.949231 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:13:35.949210    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437\": container with ID starting with 1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437 not found: ID does not exist" containerID="1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437"
Nov 02 19:13:35.949282 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.949241    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437"} err="failed to get container status \"1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437\": rpc error: code = NotFound desc = could not find container \"1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437\": container with ID starting with 1c051cde850e18d12a503f6813036101a083c94007c34abbb883c1cc2560c437 not found: ID does not exist"
Nov 02 19:13:35.949282 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.949262    4956 scope.go:122] "RemoveContainer" containerID="5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440"
Nov 02 19:13:35.949534 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:13:35.949516    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440\": container with ID starting with 5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440 not found: ID does not exist" containerID="5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440"
Nov 02 19:13:35.949591 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:35.949539    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440"} err="failed to get container status \"5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440\": rpc error: code = NotFound desc = could not find container \"5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440\": container with ID starting with 5d55da1692076ae85bed72e8a17158277dae41aab97d0d2c215e82d56c221440 not found: ID does not exist"
Nov 02 19:13:36.002055 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002021    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-socket\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002237 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002085    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-credential-socket\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002237 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002145    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-envoy\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002237 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002173    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istiod-ca-cert\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002418 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002244    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-certs\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002418 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002274    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-data\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002418 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002360    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-socket" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:13:36.002592 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002419    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-credential-socket" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:13:36.002592 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002428    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-podinfo\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002592 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002508    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-kube-api-access-kktg2\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-kube-api-access-kktg2\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002592 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002551    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-certs" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:13:36.002592 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002535    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-data" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:13:36.002592 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002566    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-token\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-token\") pod \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\" (UID: \"1ceae3d1-a28d-42d7-abb5-87ce7a9031e4\") "
Nov 02 19:13:36.002913 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002670    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istiod-ca-cert" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:13:36.002985 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002965    4956 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istiod-ca-cert\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.003239 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.002991    4956 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.003239 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.003005    4956 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-data\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.003239 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.003019    4956 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-workload-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.003239 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.003033    4956 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-credential-socket\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.004763 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.004735    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-envoy" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:13:36.004763 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.004742    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-token" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:13:36.004763 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.004755    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-podinfo" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 19:13:36.004940 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.004782    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-kube-api-access-kktg2" pod "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" (UID: "1ceae3d1-a28d-42d7-abb5-87ce7a9031e4"). InnerVolumeSpecName "kube-api-access-kktg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:13:36.104545 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.104507    4956 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-envoy\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.104545 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.104540    4956 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-podinfo\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.104545 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.104554    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktg2\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-kube-api-access-kktg2\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.104789 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.104565    4956 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4-istio-token\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:36.352215 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.352173    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"]
Nov 02 19:13:36.355899 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:36.355879    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-5b22de-openshift-default-c9dddcc57-bv9wx"]
Nov 02 19:13:37.863638 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:37.863603    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" path="/var/lib/kubelet/pods/1ceae3d1-a28d-42d7-abb5-87ce7a9031e4/volumes"
Nov 02 19:13:48.395892 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:48.395832    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:13:48.396637 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:48.396603    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:13:48.494363 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:48.494294    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:13:48.494722 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:13:48.494690    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:13:59.052749 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:13:59.052707    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:14:29.052685 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:14:29.052649    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:14:59.055372 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:14:59.055323    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:15:29.051115 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:15:29.051079    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:15:58.170602 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.170567    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"]
Nov 02 19:15:58.171026 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.170993    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy"
Nov 02 19:15:58.171026 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.171008    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy"
Nov 02 19:15:58.171110 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.171048    4956 cpu_manager.go:414] "containerMap: removing container" podUID="a9aa805b-7fa8-456a-b7d6-ec9757a83fac" containerName="maas-api"
Nov 02 19:15:58.171110 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.171056    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9aa805b-7fa8-456a-b7d6-ec9757a83fac" containerName="maas-api"
Nov 02 19:15:58.171305 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.171181    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9aa805b-7fa8-456a-b7d6-ec9757a83fac" containerName="maas-api"
Nov 02 19:15:58.171305 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.171195    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy"
Nov 02 19:15:58.174434 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.174413    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.245952 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.245914    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016702c1-4a89-41f2-b733-d0df3809dbba-openshift-service-ca\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.246174 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.245962    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6st5\" (UniqueName: \"kubernetes.io/projected/016702c1-4a89-41f2-b733-d0df3809dbba-kube-api-access-f6st5\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.246174 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.245995    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016702c1-4a89-41f2-b733-d0df3809dbba-maas-api-tls\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.246278 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.246186    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016702c1-4a89-41f2-b733-d0df3809dbba-metrics-certs\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.277342 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.277312    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"]
Nov 02 19:15:58.347524 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.347489    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016702c1-4a89-41f2-b733-d0df3809dbba-maas-api-tls\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.347740 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.347540    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016702c1-4a89-41f2-b733-d0df3809dbba-metrics-certs\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.347740 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.347615    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016702c1-4a89-41f2-b733-d0df3809dbba-openshift-service-ca\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.347740 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.347641    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6st5\" (UniqueName: \"kubernetes.io/projected/016702c1-4a89-41f2-b733-d0df3809dbba-kube-api-access-f6st5\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.348328 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.348308    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016702c1-4a89-41f2-b733-d0df3809dbba-openshift-service-ca\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.350310 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.350284    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016702c1-4a89-41f2-b733-d0df3809dbba-maas-api-tls\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.350411 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.350285    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016702c1-4a89-41f2-b733-d0df3809dbba-metrics-certs\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.403641 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.403605    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6st5\" (UniqueName: \"kubernetes.io/projected/016702c1-4a89-41f2-b733-d0df3809dbba-kube-api-access-f6st5\") pod \"maas-api-5c87d67b66-m4pkx\" (UID: \"016702c1-4a89-41f2-b733-d0df3809dbba\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.487619 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.487586    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:15:58.855816 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:58.855790    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"]
Nov 02 19:15:58.857899 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:15:58.857872    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016702c1_4a89_41f2_b733_d0df3809dbba.slice/crio-52852d588ab624e7b92aa9e1aed44c3e32a0fdae5a9489eed8afddbba81939d3 WatchSource:0}: Error finding container 52852d588ab624e7b92aa9e1aed44c3e32a0fdae5a9489eed8afddbba81939d3: Status 404 returned error can't find the container with id 52852d588ab624e7b92aa9e1aed44c3e32a0fdae5a9489eed8afddbba81939d3
Nov 02 19:15:59.052395 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:15:59.052359    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:15:59.668544 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:15:59.668464    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx" event={"ID":"016702c1-4a89-41f2-b733-d0df3809dbba","Type":"ContainerStarted","Data":"52852d588ab624e7b92aa9e1aed44c3e32a0fdae5a9489eed8afddbba81939d3"}
Nov 02 19:16:00.675560 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:00.675524    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx" event={"ID":"016702c1-4a89-41f2-b733-d0df3809dbba","Type":"ContainerStarted","Data":"a55f7e0e72d0a8c6d4747463a3238645f3a085210313462c9b7e7aa2e6051809"}
Nov 02 19:16:00.676104 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:00.675717    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:16:00.788644 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:00.788584    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx" podStartSLOduration=1.830555032 podStartE2EDuration="2.788568959s" totalImagesPullingTime="958.013927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:16:00.787822533 +0000 UTC m=+6791.044853154" watchObservedRunningTime="2026-11-02 19:16:00.788568959 +0000 UTC m=+6791.045599579"
Nov 02 19:16:06.685900 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:06.685870    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-m4pkx"
Nov 02 19:16:06.850313 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:06.850284    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"]
Nov 02 19:16:06.850523 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:06.850482    4956 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn" podUID="861c6a94-0760-4f29-92b7-6d1a23878399" containerName="maas-api" containerID="cri-o://5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3" gracePeriod=30
Nov 02 19:16:08.107916 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.107884    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 19:16:08.159531 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.159493    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-maas-api-tls\") pod \"861c6a94-0760-4f29-92b7-6d1a23878399\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") "
Nov 02 19:16:08.159712 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.159586    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861c6a94-0760-4f29-92b7-6d1a23878399-kube-api-access-nvtmb\" (UniqueName: \"kubernetes.io/projected/861c6a94-0760-4f29-92b7-6d1a23878399-kube-api-access-nvtmb\") pod \"861c6a94-0760-4f29-92b7-6d1a23878399\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") "
Nov 02 19:16:08.159712 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.159689    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-metrics-certs\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-metrics-certs\") pod \"861c6a94-0760-4f29-92b7-6d1a23878399\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") "
Nov 02 19:16:08.159789 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.159730    4956 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/861c6a94-0760-4f29-92b7-6d1a23878399-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861c6a94-0760-4f29-92b7-6d1a23878399-openshift-service-ca\") pod \"861c6a94-0760-4f29-92b7-6d1a23878399\" (UID: \"861c6a94-0760-4f29-92b7-6d1a23878399\") "
Nov 02 19:16:08.160143 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.160092    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861c6a94-0760-4f29-92b7-6d1a23878399-openshift-service-ca" pod "861c6a94-0760-4f29-92b7-6d1a23878399" (UID: "861c6a94-0760-4f29-92b7-6d1a23878399"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:16:08.161892 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.161868    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-metrics-certs" pod "861c6a94-0760-4f29-92b7-6d1a23878399" (UID: "861c6a94-0760-4f29-92b7-6d1a23878399"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:16:08.162384 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.162359    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-maas-api-tls" pod "861c6a94-0760-4f29-92b7-6d1a23878399" (UID: "861c6a94-0760-4f29-92b7-6d1a23878399"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:16:08.162477 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.162388    4956 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861c6a94-0760-4f29-92b7-6d1a23878399-kube-api-access-nvtmb" pod "861c6a94-0760-4f29-92b7-6d1a23878399" (UID: "861c6a94-0760-4f29-92b7-6d1a23878399"). InnerVolumeSpecName "kube-api-access-nvtmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:16:08.267147 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.261948    4956 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-maas-api-tls\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:08.267147 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.261987    4956 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtmb\" (UniqueName: \"kubernetes.io/projected/861c6a94-0760-4f29-92b7-6d1a23878399-kube-api-access-nvtmb\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:08.267147 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.262013    4956 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861c6a94-0760-4f29-92b7-6d1a23878399-metrics-certs\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:08.267147 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.262028    4956 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/861c6a94-0760-4f29-92b7-6d1a23878399-openshift-service-ca\") on node \"ip-10-0-129-72.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:08.712164 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.712101    4956 generic.go:356] "Generic (PLEG): container finished" podID="861c6a94-0760-4f29-92b7-6d1a23878399" containerID="5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3" exitCode=0
Nov 02 19:16:08.712336 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.712156    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn" event={"ID":"861c6a94-0760-4f29-92b7-6d1a23878399","Type":"ContainerDied","Data":"5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3"}
Nov 02 19:16:08.712336 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.712191    4956 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"
Nov 02 19:16:08.712336 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.712238    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn" event={"ID":"861c6a94-0760-4f29-92b7-6d1a23878399","Type":"ContainerDied","Data":"aaa253578a53a682d48199e4ea45cc9b3fe47e5be328763c350c0d3b5ab95b95"}
Nov 02 19:16:08.712336 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.712263    4956 scope.go:122] "RemoveContainer" containerID="5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3"
Nov 02 19:16:08.722041 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.722019    4956 scope.go:122] "RemoveContainer" containerID="5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3"
Nov 02 19:16:08.722351 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:16:08.722321    4956 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3\": container with ID starting with 5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3 not found: ID does not exist" containerID="5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3"
Nov 02 19:16:08.722408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.722363    4956 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3"} err="failed to get container status \"5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3\": rpc error: code = NotFound desc = could not find container \"5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3\": container with ID starting with 5f7688dfa50f7b0a57cc14a76b91b94a1c459c193029ff6671ee6d29a88d41a3 not found: ID does not exist"
Nov 02 19:16:08.874522 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.874490    4956 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"]
Nov 02 19:16:08.876520 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:08.876497    4956 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-p6qdn"]
Nov 02 19:16:09.863806 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:09.863772    4956 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861c6a94-0760-4f29-92b7-6d1a23878399" path="/var/lib/kubelet/pods/861c6a94-0760-4f29-92b7-6d1a23878399/volumes"
Nov 02 19:16:18.393144 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:18.391230    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:16:18.393144 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:18.391315    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:16:18.493286 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:18.493243    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:16:18.493588 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:16:18.493559    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:16:29.052715 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:16:29.052674    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:16:59.051162 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:16:59.051106    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:17:29.052588 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:17:29.052551    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:17:50.812156 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:17:50.812099    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:17:50.819739 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:17:50.819716    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:17:59.057304 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:17:59.057267    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:18:29.055196 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:18:29.055148    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:18:59.053189 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:18:59.053151    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:19:18.403935 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:19:18.403255    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:19:18.403935 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:19:18.403343    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:19:18.476264 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:19:18.476222    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:19:18.476587 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:19:18.476543    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:19:29.051329 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:19:29.051292    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:19:36.639312 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:19:36.639261    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:19:36.640025 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:19:36.639967    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:19:59.053539 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:19:59.053503    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:20:06.645358 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:20:06.645281    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-268pz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:20:06.646162 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:20:06.646092    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-268pz" podUID="0788c652-a093-414f-bcf3-c96c544ef908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:20:29.051953 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:20:29.051909    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:20:48.388148 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:20:48.387226    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:20:48.388148 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:20:48.387328    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:20:48.498673 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:20:48.498231    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:20:48.498673 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:20:48.498290    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-j8pgt" podUID="0b8ee4be-5295-401a-aa31-dc9cd699e5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:20:59.116028 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:20:59.115993    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:21:29.052099 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:21:29.052056    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:21:59.052442 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:21:59.052405    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:22:03.347839 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:03.347737    4956 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tfl27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:22:03.348496 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:03.348450    4956 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tfl27" podUID="d345262e-c751-4322-afe4-579967efeaf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:22:28.365850 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:28.365817    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5c9b9f7776-2hch9_89a58678-30c1-4403-81bd-9818ec081d07/manager/0.log"
Nov 02 19:22:29.016405 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:29.016368    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fvpqg_e85ee5b1-3fcf-4372-b108-0c7ec63511fa/manager/0.log"
Nov 02 19:22:29.052888 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:22:29.052849    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:22:31.984884 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:31.984854    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c87d67b66-m4pkx_016702c1-4a89-41f2-b733-d0df3809dbba/maas-api/0.log"
Nov 02 19:22:40.918517 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:40.918480    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87ddd77c-twqmz_0a4c16e4-d81e-4d32-a714-452dd0a06f85/kube-auth-proxy/0.log"
Nov 02 19:22:41.348438 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:41.348405    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-268pz_0788c652-a093-414f-bcf3-c96c544ef908/istio-proxy/0.log"
Nov 02 19:22:43.098497 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:43.098461    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-j8pgt_0b8ee4be-5295-401a-aa31-dc9cd699e5b6/istio-proxy/0.log"
Nov 02 19:22:43.425460 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:43.425380    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-tfl27_d345262e-c751-4322-afe4-579967efeaf4/istio-proxy/0.log"
Nov 02 19:22:44.370912 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:44.370878    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-587dbc7f89-kkwdk_f50fa52e-8904-4af5-921f-0de0f05e63db/payload-processing/0.log"
Nov 02 19:22:44.590018 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:44.589992    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d9d94667c-xzxd4_cd56f1da-de64-4541-9c26-e5ba4b221dfa/router/0.log"
Nov 02 19:22:45.699358 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:45.699332    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xqw25_917a756a-a002-4e8c-ae59-65257b63ef58/main/0.log"
Nov 02 19:22:45.923802 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:45.923774    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-p5q58_4aadf12b-de32-492e-ac42-5d9f61da5175/main/0.log"
Nov 02 19:22:46.140475 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:46.140441    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b85dq_c36888bb-ff8a-4945-8a50-5aa34718d80c/main/0.log"
Nov 02 19:22:46.353345 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:46.353317    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-s4975_9963c070-7c0d-4440-a22a-13d28a2648d9/main/0.log"
Nov 02 19:22:46.998265 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:46.998237    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-d6jxc_dd9ee5aa-c5bb-4ea4-aa9d-84ba8b8a512c/main/0.log"
Nov 02 19:22:47.236238 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:47.236209    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-vhcct_5e711d1c-6058-4345-9189-de8749615f2c/main/0.log"
Nov 02 19:22:50.841427 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:50.841391    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:22:50.850817 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:50.850789    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zhfgw_eedb70cd-4f51-41ae-acfd-6c5c2044644d/console-operator/3.log"
Nov 02 19:22:57.874819 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.874783    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-89j2x/must-gather-ktdks"]
Nov 02 19:22:57.875425 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.875404    4956 cpu_manager.go:414] "containerMap: removing container" podUID="861c6a94-0760-4f29-92b7-6d1a23878399" containerName="maas-api"
Nov 02 19:22:57.875506 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.875428    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861c6a94-0760-4f29-92b7-6d1a23878399" containerName="maas-api"
Nov 02 19:22:57.875506 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.875455    4956 cpu_manager.go:414] "containerMap: removing container" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy"
Nov 02 19:22:57.875506 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.875463    4956 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy"
Nov 02 19:22:57.875671 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.875586    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="861c6a94-0760-4f29-92b7-6d1a23878399" containerName="maas-api"
Nov 02 19:22:57.875671 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.875607    4956 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ceae3d1-a28d-42d7-abb5-87ce7a9031e4" containerName="istio-proxy"
Nov 02 19:22:57.878799 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.878778    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:57.980023 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.979996    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-89j2x\"/\"default-dockercfg-548nf\""
Nov 02 19:22:57.980276 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.980190    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89j2x\"/\"kube-root-ca.crt\""
Nov 02 19:22:57.980276 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.980226    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89j2x\"/\"openshift-service-ca.crt\""
Nov 02 19:22:57.981943 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.981926    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89j2x/must-gather-ktdks"]
Nov 02 19:22:57.983433 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.983414    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pdnvs/must-gather-qdrht"]
Nov 02 19:22:57.985104 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.985086    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2126d2b-1692-44a8-bba9-c51d48a98bd8-must-gather-output\") pod \"must-gather-ktdks\" (UID: \"d2126d2b-1692-44a8-bba9-c51d48a98bd8\") " pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:57.985271 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.985257    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvfc\" (UniqueName: \"kubernetes.io/projected/d2126d2b-1692-44a8-bba9-c51d48a98bd8-kube-api-access-fmvfc\") pod \"must-gather-ktdks\" (UID: \"d2126d2b-1692-44a8-bba9-c51d48a98bd8\") " pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:57.986978 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:57.986962    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.034155 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.034102    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/must-gather-qdrht"]
Nov 02 19:22:58.038572 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.038543    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdnvs\"/\"openshift-service-ca.crt\""
Nov 02 19:22:58.038753 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.038737    4956 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdnvs\"/\"kube-root-ca.crt\""
Nov 02 19:22:58.038856 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.038834    4956 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pdnvs\"/\"default-dockercfg-hdbkh\""
Nov 02 19:22:58.086301 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.086264    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8224a410-1b7a-4f22-8b9c-ef2c41de6ade-must-gather-output\") pod \"must-gather-qdrht\" (UID: \"8224a410-1b7a-4f22-8b9c-ef2c41de6ade\") " pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.086522 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.086336    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvfc\" (UniqueName: \"kubernetes.io/projected/d2126d2b-1692-44a8-bba9-c51d48a98bd8-kube-api-access-fmvfc\") pod \"must-gather-ktdks\" (UID: \"d2126d2b-1692-44a8-bba9-c51d48a98bd8\") " pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:58.086522 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.086420    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2126d2b-1692-44a8-bba9-c51d48a98bd8-must-gather-output\") pod \"must-gather-ktdks\" (UID: \"d2126d2b-1692-44a8-bba9-c51d48a98bd8\") " pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:58.086522 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.086457    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfq7\" (UniqueName: \"kubernetes.io/projected/8224a410-1b7a-4f22-8b9c-ef2c41de6ade-kube-api-access-6jfq7\") pod \"must-gather-qdrht\" (UID: \"8224a410-1b7a-4f22-8b9c-ef2c41de6ade\") " pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.086836 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.086815    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2126d2b-1692-44a8-bba9-c51d48a98bd8-must-gather-output\") pod \"must-gather-ktdks\" (UID: \"d2126d2b-1692-44a8-bba9-c51d48a98bd8\") " pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:58.145946 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.145859    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvfc\" (UniqueName: \"kubernetes.io/projected/d2126d2b-1692-44a8-bba9-c51d48a98bd8-kube-api-access-fmvfc\") pod \"must-gather-ktdks\" (UID: \"d2126d2b-1692-44a8-bba9-c51d48a98bd8\") " pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:58.187983 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.187944    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8224a410-1b7a-4f22-8b9c-ef2c41de6ade-must-gather-output\") pod \"must-gather-qdrht\" (UID: \"8224a410-1b7a-4f22-8b9c-ef2c41de6ade\") " pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.188195 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.188033    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfq7\" (UniqueName: \"kubernetes.io/projected/8224a410-1b7a-4f22-8b9c-ef2c41de6ade-kube-api-access-6jfq7\") pod \"must-gather-qdrht\" (UID: \"8224a410-1b7a-4f22-8b9c-ef2c41de6ade\") " pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.188361 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.188338    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8224a410-1b7a-4f22-8b9c-ef2c41de6ade-must-gather-output\") pod \"must-gather-qdrht\" (UID: \"8224a410-1b7a-4f22-8b9c-ef2c41de6ade\") " pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.193114 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.193090    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89j2x/must-gather-ktdks"
Nov 02 19:22:58.249474 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.249435    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfq7\" (UniqueName: \"kubernetes.io/projected/8224a410-1b7a-4f22-8b9c-ef2c41de6ade-kube-api-access-6jfq7\") pod \"must-gather-qdrht\" (UID: \"8224a410-1b7a-4f22-8b9c-ef2c41de6ade\") " pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.301374 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.301335    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/must-gather-qdrht"
Nov 02 19:22:58.566774 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.566745    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89j2x/must-gather-ktdks"]
Nov 02 19:22:58.569042 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:22:58.569014    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2126d2b_1692_44a8_bba9_c51d48a98bd8.slice/crio-f91ff99f0444d08111f42b4eca2f7ac6661293024469051feb681c264d4d2f13 WatchSource:0}: Error finding container f91ff99f0444d08111f42b4eca2f7ac6661293024469051feb681c264d4d2f13: Status 404 returned error can't find the container with id f91ff99f0444d08111f42b4eca2f7ac6661293024469051feb681c264d4d2f13
Nov 02 19:22:58.570959 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.570938    4956 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:22:58.674573 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:58.674547    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/must-gather-qdrht"]
Nov 02 19:22:58.676198 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:22:58.676156    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8224a410_1b7a_4f22_8b9c_ef2c41de6ade.slice/crio-ad63db111442e37d0e7e007e9aae8417ce4a681759e3db42680f4ae48c9fb6e8 WatchSource:0}: Error finding container ad63db111442e37d0e7e007e9aae8417ce4a681759e3db42680f4ae48c9fb6e8: Status 404 returned error can't find the container with id ad63db111442e37d0e7e007e9aae8417ce4a681759e3db42680f4ae48c9fb6e8
Nov 02 19:22:59.052561 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:22:59.052523    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:22:59.058848 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:59.058809    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89j2x/must-gather-ktdks" event={"ID":"d2126d2b-1692-44a8-bba9-c51d48a98bd8","Type":"ContainerStarted","Data":"f91ff99f0444d08111f42b4eca2f7ac6661293024469051feb681c264d4d2f13"}
Nov 02 19:22:59.060023 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:22:59.059992    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/must-gather-qdrht" event={"ID":"8224a410-1b7a-4f22-8b9c-ef2c41de6ade","Type":"ContainerStarted","Data":"ad63db111442e37d0e7e007e9aae8417ce4a681759e3db42680f4ae48c9fb6e8"}
Nov 02 19:23:00.069595 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:00.069553    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/must-gather-qdrht" event={"ID":"8224a410-1b7a-4f22-8b9c-ef2c41de6ade","Type":"ContainerStarted","Data":"8686b34dd8736add8e1ce18373e9145b7e79b1fa1c6fffe7364f6fc3f21c3079"}
Nov 02 19:23:01.085174 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:01.084448    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/must-gather-qdrht" event={"ID":"8224a410-1b7a-4f22-8b9c-ef2c41de6ade","Type":"ContainerStarted","Data":"c1e427dad309e8558d671bcd80315e098cf401b3b396ec816a1da29eec747492"}
Nov 02 19:23:01.201334 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:01.201114    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pdnvs/must-gather-qdrht" podStartSLOduration=3.012628481 podStartE2EDuration="4.20109302s" totalImagesPullingTime="1.188464539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:23:01.197004004 +0000 UTC m=+7211.454034634" watchObservedRunningTime="2026-11-02 19:23:01.20109302 +0000 UTC m=+7211.458123640"
Nov 02 19:23:06.116408 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:06.116368    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89j2x/must-gather-ktdks" event={"ID":"d2126d2b-1692-44a8-bba9-c51d48a98bd8","Type":"ContainerStarted","Data":"00f22b571e110f25c4adc3e5f32fc0b53c9d37c4c11bec989d215302dc6898e4"}
Nov 02 19:23:06.116972 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:06.116423    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89j2x/must-gather-ktdks" event={"ID":"d2126d2b-1692-44a8-bba9-c51d48a98bd8","Type":"ContainerStarted","Data":"89f8daf9e490f76362eb32161491ab3c2ad19f9546e798e1135535a8008dd91b"}
Nov 02 19:23:06.228867 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:06.228786    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-89j2x/must-gather-ktdks" podStartSLOduration=2.129094894 podStartE2EDuration="9.228762046s" totalImagesPullingTime="7.099667152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:23:06.227644106 +0000 UTC m=+7216.484674726" watchObservedRunningTime="2026-11-02 19:23:06.228762046 +0000 UTC m=+7216.485792667"
Nov 02 19:23:07.279291 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:07.279254    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9m4h8_64d157fc-a912-44a3-9359-23691282dda2/global-pull-secret-syncer/0.log"
Nov 02 19:23:09.105638 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:09.105606    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-czpdz_c8f53a9e-fe98-4056-8d6b-29bd6f1912cc/connection-checker/0.log"
Nov 02 19:23:09.530181 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:09.530150    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gkwvc_8a562913-9258-41b3-960e-c9dc07e2637c/connection-checker/0.log"
Nov 02 19:23:09.969439 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:09.969405    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tbpf9_d269254c-0c37-4fad-b7e4-20b88ee2c2c1/connection-checker/0.log"
Nov 02 19:23:11.033245 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:11.033206    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m7rhm_4c62335d-01f0-4c4d-a2cc-4a9c3ddc4bd5/konnectivity-agent/0.log"
Nov 02 19:23:11.054381 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:11.054338    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5c9b9f7776-2hch9_89a58678-30c1-4403-81bd-9818ec081d07/manager/0.log"
Nov 02 19:23:11.583182 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:11.583146    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fvpqg_e85ee5b1-3fcf-4372-b108-0c7ec63511fa/manager/0.log"
Nov 02 19:23:12.347733 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:12.347699    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-72.us-west-1.compute.internal_7a902dc8a4f9241e56ab9949e623f113/haproxy/0.log"
Nov 02 19:23:18.587330 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.587286    4956 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"]
Nov 02 19:23:18.590029 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.590003    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.645437 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.645391    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5c9b9f7776-2hch9_89a58678-30c1-4403-81bd-9818ec081d07/manager/0.log"
Nov 02 19:23:18.649173 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.648657    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"]
Nov 02 19:23:18.669703 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.669660    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-proc\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.669923 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.669732    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-podres\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.669923 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.669802    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-sys\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.669923 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.669826    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-lib-modules\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.669923 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.669868    4956 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nkw\" (UniqueName: \"kubernetes.io/projected/68aed032-ee56-4bfb-b866-32d10347f2e6-kube-api-access-s7nkw\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771310 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771252    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-proc\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771330    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-podres\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771374    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-sys\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771393    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-proc\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771407    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-lib-modules\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771439    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-podres\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771463    4956 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nkw\" (UniqueName: \"kubernetes.io/projected/68aed032-ee56-4bfb-b866-32d10347f2e6-kube-api-access-s7nkw\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771470    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-lib-modules\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.771521 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.771496    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68aed032-ee56-4bfb-b866-32d10347f2e6-sys\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.827854 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.827820    4956 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nkw\" (UniqueName: \"kubernetes.io/projected/68aed032-ee56-4bfb-b866-32d10347f2e6-kube-api-access-s7nkw\") pod \"perf-node-gather-daemonset-88bsw\" (UID: \"68aed032-ee56-4bfb-b866-32d10347f2e6\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:18.905094 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:18.905010    4956 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:19.099751 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:19.099719    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fvpqg_e85ee5b1-3fcf-4372-b108-0c7ec63511fa/manager/0.log"
Nov 02 19:23:19.275163 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:19.275098    4956 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"]
Nov 02 19:23:19.278310 ip-10-0-129-72 kubenswrapper[4956]: W1102 19:23:19.278280    4956 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68aed032_ee56_4bfb_b866_32d10347f2e6.slice/crio-8de58484112c9b414c24a2bb5c705987b1aead3c2b8a86507d30752d98f2418e WatchSource:0}: Error finding container 8de58484112c9b414c24a2bb5c705987b1aead3c2b8a86507d30752d98f2418e: Status 404 returned error can't find the container with id 8de58484112c9b414c24a2bb5c705987b1aead3c2b8a86507d30752d98f2418e
Nov 02 19:23:20.242759 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:20.242695    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw" event={"ID":"68aed032-ee56-4bfb-b866-32d10347f2e6","Type":"ContainerStarted","Data":"2dc98ed0a3a93e13889a9d5d72c90dcb763c680096999ab2bb695dc25de2cbdb"}
Nov 02 19:23:20.243226 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:20.242824    4956 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw" event={"ID":"68aed032-ee56-4bfb-b866-32d10347f2e6","Type":"ContainerStarted","Data":"8de58484112c9b414c24a2bb5c705987b1aead3c2b8a86507d30752d98f2418e"}
Nov 02 19:23:20.243226 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:20.242859    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:20.355489 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:20.355430    4956 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw" podStartSLOduration=2.355411445 podStartE2EDuration="2.355411445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:23:20.354647282 +0000 UTC m=+7230.611677902" watchObservedRunningTime="2026-11-02 19:23:20.355411445 +0000 UTC m=+7230.612442065"
Nov 02 19:23:25.716130 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:25.716080    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5c9b9f7776-2hch9_89a58678-30c1-4403-81bd-9818ec081d07/manager/0.log"
Nov 02 19:23:26.249092 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:26.249058    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fvpqg_e85ee5b1-3fcf-4372-b108-0c7ec63511fa/manager/0.log"
Nov 02 19:23:26.263245 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:26.263209    4956 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-88bsw"
Nov 02 19:23:26.539414 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:26.539327    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ghflt_fb5d1d52-3223-45e9-a766-848d837c029b/cluster-monitoring-operator/0.log"
Nov 02 19:23:29.067942 ip-10-0-129-72 kubenswrapper[4956]: E1102 19:23:29.066081    4956 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n69x" containerName="extractor"
Nov 02 19:23:30.301086 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:30.301002    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5x8kk_20ad288b-fc96-4d66-9c2c-336ae6e60afb/node-exporter/0.log"
Nov 02 19:23:30.716323 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:30.716281    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5x8kk_20ad288b-fc96-4d66-9c2c-336ae6e60afb/kube-rbac-proxy/0.log"
Nov 02 19:23:31.133331 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:31.133257    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5x8kk_20ad288b-fc96-4d66-9c2c-336ae6e60afb/init-textfile/0.log"
Nov 02 19:23:35.654861 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:35.654826    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87ddd77c-twqmz_0a4c16e4-d81e-4d32-a714-452dd0a06f85/kube-auth-proxy/0.log"
Nov 02 19:23:36.090988 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:36.090940    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-268pz_0788c652-a093-414f-bcf3-c96c544ef908/istio-proxy/0.log"
Nov 02 19:23:37.590836 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:37.590801    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-j8pgt_0b8ee4be-5295-401a-aa31-dc9cd699e5b6/istio-proxy/0.log"
Nov 02 19:23:37.806272 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:37.806243    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-tfl27_d345262e-c751-4322-afe4-579967efeaf4/istio-proxy/0.log"
Nov 02 19:23:38.654259 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:38.654223    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-587dbc7f89-kkwdk_f50fa52e-8904-4af5-921f-0de0f05e63db/payload-processing/0.log"
Nov 02 19:23:38.876215 ip-10-0-129-72 kubenswrapper[4956]: I1102 19:23:38.876186    4956 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d9d94667c-xzxd4_cd56f1da-de64-4541-9c26-e5ba4b221dfa/router/0.log"